Skip to content

Commit

Permalink
Fix SDXL readme and submission tool (#1798)
Browse files Browse the repository at this point in the history
  • Loading branch information
nvyihengz authored Jul 22, 2024
1 parent 2814499 commit a248595
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 4 deletions.
4 changes: 2 additions & 2 deletions compliance/nvidia/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ This repository provides the compliance tests that need to be run by the submitt
## Introduction
The purpose of compliance testing is to ensure a basic level of compliance with a subset of the MLPerf rules. The tests are designed to be complementary to third-party auditing which will be introduced in future rounds of MLPerf. The tests are not meant to root-cause issues with the submission, but can help detect anomalies in the submission that need to be investigated further by the submitter.

Each compliance test must be run once for each submission run and the logs from the compliance test run must be uploaded along with the rest of the submission collateral. In MLPerf Inference v0.7, effort has been made to reduce the burden on submitters to perform compliance testing through improvements to documentation, scripting, and LoadGen's compliance functionality. More documentation is provided on the purpose of each test in the corresponding test directory, along with more detailed instructions.
Each compliance test must be run once for each submission run and the logs from the compliance test run must be uploaded along with the rest of the submission collateral. In MLPerf Inference v0.7, effort has been made to reduce the burden on submitters to perform compliance testing through improvements to documentation, scripting, and LoadGen's compliance functionality. More documentation is provided on the purpose of each test in the corresponding test directory, along with more detailed instructions.

## Test Infrastructure
The compliance tests exercise functionality in LoadGen, triggered through the use of a config file that overrides LoadGen functionality. This enables LoadGen to run in a variety of compliance testing modes. When LoadGen::StartTest() is invoked, LoadGen checks if a `audit.config` file exists in the current working directory. If the file is found, LoadGen will log this event in `mlperf_log_detail.txt`. The LoadGen settings that are used will be logged in `mlperf_log_summary.txt`. The configuration parameters in `audit.config` override any settings set by `mlperf.conf` or `user.conf`.
Expand All @@ -37,6 +37,6 @@ The `run_verification.py` found in each test directory will copy the test files
| 3d-unet | [TEST01](./TEST01/), [TEST05](./TEST05/) |
| rnnt | [TEST01](./TEST01/), [TEST05](./TEST05/) |
| gpt-j | - |
| stable-diffusion-xl | [TEST01](./TEST01/), [TEST04](./TEST04/), [TEST05](./TEST05/) |
| stable-diffusion-xl | [TEST01](./TEST01/), [TEST04](./TEST04/) |
| Llama2-70b | [TEST06](./TEST06/) |
| mixtral-8x7b | [TEST06](./TEST06/) |
6 changes: 4 additions & 2 deletions tools/submission/generate_final_report.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ def main():
lambda x: '=HYPERLINK("{}","details")'.format('/'.join(
[base_url, x['Category'], x['Submitter'], 'results', x['Platform']])),
axis=1)

# code url
df['Code'] = df.apply(
lambda x: '=HYPERLINK("{}","code")'.format('/'.join(
Expand Down Expand Up @@ -104,7 +104,8 @@ def main():
[
'resnet', 'retinanet', '3d-unet-99', '3d-unet-99.9',
'rnnt', 'bert-99', 'bert-99.9', 'dlrm-v2-99', 'dlrm-v2-99.9',
'gptj-99', 'gptj-99.9', 'stable-diffusion-xl', 'llama2-70b-99', 'llama2-70b-99.9'
'gptj-99', 'gptj-99.9', 'stable-diffusion-xl', 'llama2-70b-99', 'llama2-70b-99.9',
'mixtral-8x7b',
], ['SingleStream', 'MultiStream', 'Server', 'Offline'],
[
'Latency (ms)',
Expand All @@ -130,6 +131,7 @@ def main():
'stable-diffusion-xl': ['Server', 'Offline'],
'llama2-70b-99': ['Server', 'Offline'],
'llama2-70b-99.9': ['Server', 'Offline'],
'mixtral-8x7b': ['Server', 'Offline'],
},
'edge': {
'resnet': ['SingleStream', 'MultiStream', 'Offline'],
Expand Down

0 comments on commit a248595

Please sign in to comment.