Updated docs

mlcommons · Apr 10, 2024 · a173a4f · a173a4f
1 parent ac5936e
commit a173a4f
Show file tree

Hide file tree

Showing 2 changed files with 62 additions and 9 deletions.
diff --git a/cm-mlops/script/app-loadgen-generic-python/README.md b/cm-mlops/script/app-loadgen-generic-python/README.md
@@ -54,7 +54,10 @@ Developers: [Gaz Iqbal](https://www.linkedin.com/in/gaziqbal), [Arjun Suresh](ht
 
 #### Input Flags
 
-* --**modelpath**=Full path to a model
+* --**modelpath**=Full path to file with model weights
+* --**modelcodepath**=(for PyTorch models) Full path to file with model code and cmc.py
+* --**modelcfgpath**=(for PyTorch models) Full path to JSON file with model cfg
+* --**modelsamplepath**=(for PyTorch models) Full path to file with model sample in pickle format
 * --**ep**=ONNX Execution provider
 * --**scenario**=MLPerf LoadGen scenario
 * --**samples**=Number of samples (*2*)
@@ -116,6 +119,15 @@ ___
     <details>
     <summary>Click here to expand this section.</summary>
 
+    * `_cmc`
+      - Environment variables:
+        - *CM_CUSTOM_MODEL_CMC*: `True`
+      - Workflow:
+    * `_custom,cmc`
+      - Workflow:
+        1. ***Read "deps" on other CM scripts***
+           * get,ml-model,cmc
+             - *Warning: no scripts found*
     * `_custom,huggingface`
       - Workflow:
         1. ***Read "deps" on other CM scripts***
@@ -201,7 +213,13 @@ ___
 * `--execmode=value`  &rarr;  `CM_MLPERF_EXEC_MODE=value`
 * `--interop=value`  &rarr;  `CM_MLPERF_INTEROP=value`
 * `--intraop=value`  &rarr;  `CM_MLPERF_INTRAOP=value`
+* `--loadgen_duration_sec=value`  &rarr;  `CM_MLPERF_LOADGEN_DURATION_SEC=value`
+* `--loadgen_expected_qps=value`  &rarr;  `CM_MLPERF_LOADGEN_EXPECTED_QPS=value`
+* `--modelcfg=value`  &rarr;  `CM_ML_MODEL_CFG=value`
+* `--modelcfgpath=value`  &rarr;  `CM_ML_MODEL_CFG_WITH_PATH=value`
+* `--modelcodepath=value`  &rarr;  `CM_ML_MODEL_CODE_WITH_PATH=value`
 * `--modelpath=value`  &rarr;  `CM_ML_MODEL_FILE_WITH_PATH=value`
+* `--modelsamplepath=value`  &rarr;  `CM_ML_MODEL_SAMPLE_WITH_PATH=value`
 * `--output_dir=value`  &rarr;  `CM_MLPERF_OUTPUT_DIR=value`
 * `--runner=value`  &rarr;  `CM_MLPERF_RUNNER=value`
 * `--samples=value`  &rarr;  `CM_MLPERF_LOADGEN_SAMPLES=value`
@@ -241,6 +259,8 @@ ___
        - CM script: [get-python3](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-python3)
      * get,generic-python-lib,_psutil
        - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
+     * get,generic-python-lib,_package.numpy
+       - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
      * get,cuda
        * `if (CM_MLPERF_DEVICE  == gpu)`
        - CM script: [get-cuda](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cuda)
@@ -259,6 +279,22 @@ ___
        * `if (CM_MLPERF_BACKEND  == onnxruntime)`
        * CM names: `--adr.['onnx']...`
        - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
+     * get,generic-python-lib,_torch
+       * `if (CM_MLPERF_BACKEND  == pytorch AND CM_MLPERF_DEVICE  == cpu)`
+       * CM names: `--adr.['torch']...`
+       - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
+     * get,generic-python-lib,_torchvision
+       * `if (CM_MLPERF_BACKEND  == pytorch AND CM_MLPERF_DEVICE  == cpu)`
+       * CM names: `--adr.['torchvision']...`
+       - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
+     * get,generic-python-lib,_torch_cuda
+       * `if (CM_MLPERF_BACKEND  == pytorch AND CM_MLPERF_DEVICE  == gpu)`
+       * CM names: `--adr.['torch']...`
+       - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
+     * get,generic-python-lib,_torchvision_cuda
+       * `if (CM_MLPERF_BACKEND  == pytorch AND CM_MLPERF_DEVICE  == gpu)`
+       * CM names: `--adr.['torchvision']...`
+       - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
      * get,ml-model,resnet50,_onnx
        * `if (CM_MODEL  == resnet50)`
        - CM script: [get-ml-model-resnet50](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-resnet50)

diff --git a/cm-mlops/script/get-generic-python-lib/README.md b/cm-mlops/script/get-generic-python-lib/README.md
@@ -40,13 +40,13 @@ Maintainers: [Public MLCommons Task Force on Automation and Reproducibility](htt
 
 `cm run script --tags=get,generic-python-lib`
 
-`cm run script --tags=get,generic-python-lib[,variations] `
+`cm run script --tags=get,generic-python-lib[,variations] [--input_flags]`
 
 *or*
 
 `cmr "get generic-python-lib"`
 
-`cmr "get generic-python-lib [variations]" `
+`cmr "get generic-python-lib [variations]" [--input_flags]`
 
 
 * *See the list of `variations` [here](#variations) and check the [Gettings Started Guide](https://github.com/mlcommons/ck/blob/dev/docs/getting-started.md) for more details.*
@@ -85,7 +85,7 @@ Use this [online GUI](https://cKnowledge.org/cm-gui/?tags=get,generic-python-lib
 
 #### Run this script via Docker (beta)
 
-`cm docker script "get generic-python-lib[variations]" `
+`cm docker script "get generic-python-lib[variations]" [--input_flags]`
 
 ___
 ### Customization
@@ -487,8 +487,8 @@ ___
       - Workflow:
     * `_torch,rocm`
       - Environment variables:
-        - *CM_GENERIC_PYTHON_PIP_INDEX_URL*: `https://download.pytorch.org/whl/rocm5.6`
         - *CM_GENERIC_PYTHON_PACKAGE_NAME*: `torch`
+        - *CM_GENERIC_PYTHON_PIP_INDEX_URL*: `https://download.pytorch.org/whl/rocm5.6`
         - *CM_GENERIC_PYTHON_PIP_UNINSTALL_DEPS*: `torch`
       - Workflow:
         1. ***Read "post_deps" on other CM scripts***
@@ -509,8 +509,8 @@ ___
     * `_torch_cuda,pre`
       - Environment variables:
         - *CM_GENERIC_PYTHON_PACKAGE_NAME*: `torch`
-        - *CM_GENERIC_PYTHON_PIP_INDEX_URL*: `https://download.pytorch.org/whl/<<<CM_CUDA_VERSION_STRING>>>`
         - *CM_GENERIC_PYTHON_PIP_EXTRA*: ` --pre`
+        - *CM_GENERIC_PYTHON_PIP_INDEX_URL*: `https://download.pytorch.org/whl/<<<CM_CUDA_VERSION_STRING>>>`
         - *CM_TORCH_VERSION_EXTRA*: `CUDA`
       - Workflow:
         1. ***Read "deps" on other CM scripts***
@@ -532,8 +532,8 @@ ___
       - Workflow:
     * `_torchaudio,rocm`
       - Environment variables:
-        - *CM_GENERIC_PYTHON_PIP_INDEX_URL*: `https://download.pytorch.org/whl/rocm5.6`
         - *CM_GENERIC_PYTHON_PACKAGE_NAME*: `torchaudio`
+        - *CM_GENERIC_PYTHON_PIP_INDEX_URL*: `https://download.pytorch.org/whl/rocm5.6`
         - *CM_GENERIC_PYTHON_PIP_UNINSTALL_DEPS*: `torchaudio`
       - Workflow:
     * `_torchaudio_cuda`
@@ -553,8 +553,8 @@ ___
       - Workflow:
     * `_torchvision,rocm`
       - Environment variables:
-        - *CM_GENERIC_PYTHON_PIP_INDEX_URL*: `https://download.pytorch.org/whl/rocm5.6`
         - *CM_GENERIC_PYTHON_PACKAGE_NAME*: `torchvision`
+        - *CM_GENERIC_PYTHON_PIP_INDEX_URL*: `https://download.pytorch.org/whl/rocm5.6`
         - *CM_GENERIC_PYTHON_PIP_UNINSTALL_DEPS*: `torchvision`
       - Workflow:
     * `_torchvision_cuda`
@@ -614,6 +614,23 @@ ___
 
     </details>
 
+
+#### Script flags mapped to environment
+<details>
+<summary>Click here to expand this section.</summary>
+
+* `--extra_index_url=value`  &rarr;  `CM_GENERIC_PYTHON_PIP_EXTRA_INDEX_URL=value`
+* `--force_install=value`  &rarr;  `CM_TMP_PYTHON_PACKAGE_FORCE_INSTALL=value`
+* `--index_url=value`  &rarr;  `CM_GENERIC_PYTHON_PIP_INDEX_URL=value`
+
+**Above CLI flags can be used in the Python CM API as follows:**
+
+```python
+r=cm.access({... , "extra_index_url":...}
+```
+
+</details>
+
 #### Default environment
 
 <details>
@@ -655,7 +672,7 @@ ___
 
 ___
 ### Script output
-`cmr "get generic-python-lib [,variations]"  -j`
+`cmr "get generic-python-lib [,variations]" [--input_flags] -j`
 #### New environment keys (filter)
 
 * `CM_PYTHONLIB_*`