New front end #1947
New front end #1947
Conversation
…rfm/helm into farzaan/new-front-end
src/helm-frontend/public/config.js
Outdated
| window.BENCHMARK_OUTPUT_BASE_URL = "benchmark_output"; | ||
| window.SUITE = "latest"; | ||
| window.RELEASE = null; | ||
| window.LEGACY_RELEASE = "v2.4.0"; |
There was a problem hiding this comment.
Where is LEGACY_RELEASE used?
| @@ -3,7 +3,7 @@ export default function Card() { | |||
| <div className="stats shadow"> | |||
| <div className="stat"> | |||
| <div className="stat-title">Total Models</div> | |||
| <div className="stat-value">123</div> | |||
| <div className="stat-value">1</div> | |||
src/helm-frontend/index.html
Outdated
There was a problem hiding this comment.
Can we also add the Google Analytics tracker here (commented out)?
| {" "} | ||
| a transparent benchmarking system for language models{" "} |
There was a problem hiding this comment.
Delete both {" "} instead <strong>.
| window.SUITE = "v0.2.4"; | ||
| window.RELEASE = "v0.3.0"; |
There was a problem hiding this comment.
Either suite or release should be set, not both. See my other comment later on the semantics of this.
| .map((subGroup, idx) => | ||
| subGroup.todo || subGroup.name.includes("CLEVA") ? ( |
There was a problem hiding this comment.
We should just set todo to true for all the CLEVA results and then remove this .includes() bit.
| @@ -17,5 +17,5 @@ export default defineConfig({ | |||
| build: { | |||
| outDir: `${__dirname}/../helm/benchmark/static_build`, | |||
| }, | |||
| base: "/helm/", | |||
| //base: "/helm/" // can't add process.env.HELM_SUITE here with GH pages | |||
There was a problem hiding this comment.
Could you add some documentation somewhere about how to configure and build the website with these changes e.g. does this have to change to /helm/latest/ before building?
There was a problem hiding this comment.
A comment in the code would be sufficient for this, for now.
| if (window.RELEASE) { | ||
| return `/benchmark_output/releases/${window.RELEASE}`; | ||
| } else { | ||
| return `/benchmark_output/runs/${window.SUITE}`; |
There was a problem hiding this comment.
This needs to change to reflect the mechanics of the incremental release system:
JSON files pertaining to aggregations across multiple runs are stored in /benchmark_output/releases/v0.3.1/... e.g. /benchmark_output/releases/v0.3.1/groups.json
JSON files pertaining to individual runs are stored in suites /benchmark_output/runs/v0.3.0/run_name/... - but a single release can have runs from multiple suites, so there could also be runs from /benchmark_output/runs/v0.3.0/run_name/..., /benchmark_output/runs/v0.3.1/run_name/... etc.
For each run name, you need to look at /benchmark_output/runs/v0.3.0/runs_to_run_suites.json file to figure out which suite each run is from (example).
For instance, in the legacy code, we load runs_to_run_suites here, then we compute the URLs here and here.
This also means that computing the URL is an async function.
In the old code, we also supported a legacy mode, where if window.SUITE is set but window.RELEASE is not set (only one of these two will be set), then all files are stored in the suites e.g. /benchmark_output/runs/v0.3.1/groups.json, and runs_to_run_suites.json does not exist because all runs are in the same suite. I think we can punt on this for now... I'm not sure how much legacy stuff we want to support.
Currently live at farzaank.github.io/helm
Introduces the following changes:
Future PR will make Leaderboard the landing page, and moving things around as a result