diff --git a/compliance/nvidia/README.md b/compliance/nvidia/README.md index 630676180..3abb256c1 100755 --- a/compliance/nvidia/README.md +++ b/compliance/nvidia/README.md @@ -10,7 +10,7 @@ This repository provides the compliance tests that need to be run by the submitt ## Introduction The purpose of compliance testing is to ensure a basic level of compliance with a subset of the MLPerf rules. The tests are designed to be complementary to third-party auditing which will be introduced in future rounds of MLPerf. The tests are not meant to root-cause issues with the submission, but can help detect anomalies in the submission that need to be investigated further by the submitter. -Each compliance test must be run once for each submission run and the logs from the compliance test run must be uploaded along with the rest of the submission collateral. In MLPerf Inference v0.7, effort has been made to reduce the burden on submitters to perform compliance testing through improvements to documentation, scripting, and LoadGen's compliance functionality. More documentation is provided on the purpose of each test in the corresponding test directory, along with more detailed instructions. +Each compliance test must be run once for each submission run and the logs from the compliance test run must be uploaded along with the rest of the submission collateral. In MLPerf Inference v0.7, effort has been made to reduce the burden on submitters to perform compliance testing through improvements to documentation, scripting, and LoadGen's compliance functionality. More documentation is provided on the purpose of each test in the corresponding test directory, along with more detailed instructions. ## Test Infrastructure The compliance tests exercise functionality in LoadGen, triggered through the use of a config file that overrides LoadGen functionality. This enables LoadGen to run in a variety of compliance testing modes. When LoadGen::StartTest() is invoked, LoadGen checks if a `audit.config` file exists in the current working directory. If the file is found, LoadGen will log this event in `mlperf_log_detail.txt`. The LoadGen settings that are used will be logged in `mlperf_log_summary.txt`. The configuration parameters in `audit.config` override any settings set by `mlperf.conf` or `user.conf`. @@ -37,6 +37,6 @@ The `run_verification.py` found in each test directory will copy the test files | 3d-unet | [TEST01](./TEST01/), [TEST05](./TEST05/) | | rnnt | [TEST01](./TEST01/), [TEST05](./TEST05/) | | gpt-j | - | -| stable-diffusion-xl | [TEST01](./TEST01/), [TEST04](./TEST04/), [TEST05](./TEST05/) | +| stable-diffusion-xl | [TEST01](./TEST01/), [TEST04](./TEST04/) | | Llama2-70b | [TEST06](./TEST06/) | | mixtral-8x7b | [TEST06](./TEST06/) | diff --git a/tools/submission/generate_final_report.py b/tools/submission/generate_final_report.py index 13a01fe55..a39098b73 100644 --- a/tools/submission/generate_final_report.py +++ b/tools/submission/generate_final_report.py @@ -61,7 +61,7 @@ def main(): lambda x: '=HYPERLINK("{}","details")'.format('/'.join( [base_url, x['Category'], x['Submitter'], 'results', x['Platform']])), axis=1) - + # code url df['Code'] = df.apply( lambda x: '=HYPERLINK("{}","code")'.format('/'.join( @@ -104,7 +104,8 @@ def main(): [ 'resnet', 'retinanet', '3d-unet-99', '3d-unet-99.9', 'rnnt', 'bert-99', 'bert-99.9', 'dlrm-v2-99', 'dlrm-v2-99.9', - 'gptj-99', 'gptj-99.9', 'stable-diffusion-xl', 'llama2-70b-99', 'llama2-70b-99.9' + 'gptj-99', 'gptj-99.9', 'stable-diffusion-xl', 'llama2-70b-99', 'llama2-70b-99.9', + 'mixtral-8x7b', ], ['SingleStream', 'MultiStream', 'Server', 'Offline'], [ 'Latency (ms)', @@ -130,6 +131,7 @@ def main(): 'stable-diffusion-xl': ['Server', 'Offline'], 'llama2-70b-99': ['Server', 'Offline'], 'llama2-70b-99.9': ['Server', 'Offline'], + 'mixtral-8x7b': ['Server', 'Offline'], }, 'edge': { 'resnet': ['SingleStream', 'MultiStream', 'Offline'],