From a46164678667ca7ccd8764cd7c80fdd35751e596 Mon Sep 17 00:00:00 2001 From: anandhu-eng Date: Tue, 20 Aug 2024 11:09:31 +0530 Subject: [PATCH 1/4] added info about hf password --- docs/benchmarks/language/get-llama2-70b-data.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/benchmarks/language/get-llama2-70b-data.md b/docs/benchmarks/language/get-llama2-70b-data.md index 917f4705e..bdbb97b87 100644 --- a/docs/benchmarks/language/get-llama2-70b-data.md +++ b/docs/benchmarks/language/get-llama2-70b-data.md @@ -28,4 +28,8 @@ Get the Official MLPerf LLAMA2-70b Model ``` cm run script --tags=get,ml-model,llama2-70b,_pytorch -j ``` + +!!! tip + + Downloading llama2-70B model from Hugging Face will prompt you to enter the Hugging Face username and password. Please note that the password required is the **access token** generated for your account. Additionally, ensure that your account has access to the llama2-70B model. From dd47e451e59005aaf90bdf6f4c0d56da614568fe Mon Sep 17 00:00:00 2001 From: anandhu-eng Date: Tue, 20 Aug 2024 14:14:13 +0530 Subject: [PATCH 2/4] added links to model and access tokens --- docs/benchmarks/language/get-llama2-70b-data.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/benchmarks/language/get-llama2-70b-data.md b/docs/benchmarks/language/get-llama2-70b-data.md index bdbb97b87..0214d95a5 100644 --- a/docs/benchmarks/language/get-llama2-70b-data.md +++ b/docs/benchmarks/language/get-llama2-70b-data.md @@ -31,5 +31,5 @@ Get the Official MLPerf LLAMA2-70b Model !!! tip - Downloading llama2-70B model from Hugging Face will prompt you to enter the Hugging Face username and password. Please note that the password required is the **access token** generated for your account. Additionally, ensure that your account has access to the llama2-70B model. + Downloading llama2-70B model from Hugging Face will prompt you to enter the Hugging Face username and password. Please note that the password required is the [**access token**](https://huggingface.co/settings/tokens) generated for your account. Additionally, ensure that your account has access to the [llama2-70B](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) model. From a1d66d4ef66fbf15de8ef0fc7270d2dd06d3e52f Mon Sep 17 00:00:00 2001 From: anandhu-eng Date: Tue, 20 Aug 2024 17:06:04 +0530 Subject: [PATCH 3/4] Updated reference results structuree tree --- docs/submission/index.md | 31 ++++++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/docs/submission/index.md b/docs/submission/index.md index 9e6b66044..4c892379b 100644 --- a/docs/submission/index.md +++ b/docs/submission/index.md @@ -12,20 +12,37 @@ hide: === "Non CM based benchmark" If you have not followed the `cm run` commands under the individual model pages in the [benchmarks](../index.md) directory, please make sure that the result directory is structured in the following way. ``` - └── SUT_Name + └── System description ID ├── cm_sut_info.json - └── model + └── Benchmark └── Scenario - └── loadgen_Mode - ├── file-1 - ├── ... - └── file-n + ├── Performance + | └── run_x/#1 run for all scenarios + | ├── mlperf_log_summary.txt + | └── mlperf_log_detail.txt + ├── Accuracy + | ├── mlperf_log_summary.txt + | ├── mlperf_log_detail.txt + | ├── mlperf_log_accuracy.json + | └── accuracy.txt + └── Test_ID + ├── Performance + | └── run_x/#1 run for all scenarios + | ├── mlperf_log_summary.txt + | └── mlperf_log_detail.txt + ├── Accuracy + | ├── baseline_accuracy.txt + | ├── compliance_accuracy.txt + | ├── mlperf_log_accuracy.json + | └── accuracy.txt + ├── verify_performance.txt + └── verify_accuracy.txt #for TEST01 only ``` !!! tip - The `cm_sut_info.json` should contain the following keys - - `system_name` + - `hardware_name` - `implementation` - `device` - `framework` From c5ae6ed9ae6877a99efc657a64664e426ba9e6f2 Mon Sep 17 00:00:00 2001 From: anandhu-eng Date: Tue, 20 Aug 2024 18:23:20 +0530 Subject: [PATCH 4/4] submission docs cleanup --- docs/submission/index.md | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/docs/submission/index.md b/docs/submission/index.md index 4c892379b..a75bc3259 100644 --- a/docs/submission/index.md +++ b/docs/submission/index.md @@ -12,8 +12,8 @@ hide: === "Non CM based benchmark" If you have not followed the `cm run` commands under the individual model pages in the [benchmarks](../index.md) directory, please make sure that the result directory is structured in the following way. ``` - └── System description ID - ├── cm_sut_info.json + └── System description ID(SUT Name) + ├── system_meta.json └── Benchmark └── Scenario ├── Performance @@ -25,7 +25,7 @@ hide: | ├── mlperf_log_detail.txt | ├── mlperf_log_accuracy.json | └── accuracy.txt - └── Test_ID + └── Compliance_Test_ID ├── Performance | └── run_x/#1 run for all scenarios | ├── mlperf_log_summary.txt @@ -38,18 +38,9 @@ hide: ├── verify_performance.txt └── verify_accuracy.txt #for TEST01 only ``` - - !!! tip - - - The `cm_sut_info.json` should contain the following keys - - `hardware_name` - - `implementation` - - `device` - - `framework` - - `run_config`
- Click here if you are submitting custom model in open division + Click here if you are submitting in open division * The `model_mapping.json` should be included inside the SUT folder which is used to map the custom model full name to the official model name. The format of json file is: