2022-2024 MLCommons
Collective Mind (CM) is a collection of portable, extensible, technology-agnostic and ready-to-use automation recipes with a human-friendly interface (aka CM scripts) to unify and automate all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications on any platform with any software and hardware: see online catalog and source code.
CM scripts require Python 3.7+ with minimal dependencies and are continuously extended by the community and MLCommons members to run natively on Ubuntu, MacOS, Windows, RHEL, Debian, Amazon Linux and any other operating system, in a cloud or inside automatically generated containers while keeping backward compatibility - please don't hesitate to report encountered issues here and contact us via public Discord Server to help this collaborative engineering effort!
CM scripts were originally developed based on the following requirements from the MLCommons members to help them automatically compose and optimize complex MLPerf benchmarks, applications and systems across diverse and continuously changing models, data sets, software and hardware from Nvidia, Intel, AMD, Google, Qualcomm, Amazon and other vendors:
- must work out of the box with the default options and without the need to edit some paths, environment variables and configuration files;
- must be non-intrusive, easy to debug and must reuse existing user scripts and automation tools (such as cmake, make, ML workflows, python poetry and containers) rather than substituting them;
- must have a very simple and human-friendly command line with a Python API and minimal dependencies;
- must require minimal or zero learning curve by using plain Python, native scripts, environment variables and simple JSON/YAML descriptions instead of inventing new workflow languages;
- must have the same interface to run all automations natively, in a cloud or inside containers.
CM scripts were successfully validated by MLCommons to modularize MLPerf inference benchmarks and help the community automate more than 95% of all performance and power submissions in the v3.1 round across more than 120 system configurations (models, frameworks, hardware) while reducing development and maintenance costs.
For example, you should be able to run the MLPerf inference benchmark on Linux, Windows and MacOS using a few CM commands:
pip install cmind -U
cm pull repo mlcommons@ck
cm rm cache -f
cm run script "get mlcommons inference src"
cm run script "get generic-python-lib _onnxruntime" --version=1.17.1
cm run script "get ml-model image-classification resnet50 raw _fp32 _onnx"
cm run script "get dataset image-classification imagenet preprocessed _NCHW"
cm show cache
cm run script "run-mlperf-inference _r4.0 _performance-only _short" \
--device=cpu \
--model=resnet50 \
--precision=float32 \
--implementation=reference \
--backend=onnxruntime \
--scenario=Offline \
--clean \
--quiet \
--time
cm run script "run-mlperf-inference _r4.0 _accuracy-only _short" \
--device=cpu \
--model=resnet50 \
--precision=float32 \
--implementation=reference \
--backend=onnxruntime \
--scenario=Offline \
--clean \
--quiet \
--time
cm run script "run-mlperf-inference _r4.0 _submission _short" \
--device=cpu \
--model=resnet50 \
--precision=float32 \
--implementation=reference \
--backend=onnxruntime \
--scenario=Offline \
--clean \
--quiet \
--time
...
+----------+----------+----------+--------+-----------------+---------------------------------+
| Model | Scenario | Accuracy | QPS | Latency (in ms) | Power Efficiency (in samples/J) |
+----------+----------+----------+--------+-----------------+---------------------------------+
| resnet50 | Offline | 80.0 | 27.371 | - | |
+----------+----------+----------+--------+-----------------+---------------------------------+
The MLPerf inference results are stored at /home/gfursin/CM/repos/local/cache/a504fb143e97452f/test_results
Path to the MLPerf inference benchmark reference sources: /home/gfursin/CM/repos/local/cache/8061c243b8b54a3b/inference
Path to the MLPerf inference reference configuration file: /home/gfursin/CM/repos/local/cache/8061c243b8b54a3b/inference/mlperf.conf
You can also run the same commands using a unified CM Python API:
import cmind
output=cmind.access({
'action':'run', 'automation':'script',
'tags':'run-mlperf-inference,_r4.0,_performance-only,_short',
'device':'cpu',
'model':'resnet50',
'precision':'float32',
'implementation':'reference',
'backend':'onnxruntime',
'scenario':'Offline',
'clean':True,
'quiet':True,
'time':True,
'out':'con'
})
if output['return']==0: print (output)We suggest you to use this online CM interface to generate CM commands that will prepare and run MLPerf benchmarks and AI applications across different platforms.
Click to see more examples of CM scripts and workflows to download Stable Diffusion, GPT-J and LLAMA2 models, process datasets, install tools and compose AI applications
pip install cmind -U
cm pull repo mlcommons@ck
cm checkout repo mlcommons@ck --branch=dev
cm checkout repo mlcommons@ck --branch=master
cm rm cache -f
cm run script "python app image-classification onnx"
cmr "python app image-classification onnx"
cmr "download file _wget" --url=https://cKnowledge.org/ai/data/computer_mouse.jpg --verify=no --env.CM_DOWNLOAD_CHECKSUM=45ae5c940233892c2f860efdf0b66e7e
cmr "python app image-classification onnx" --input=computer_mouse.jpg
cmr "python app image-classification onnx" --input=computer_mouse.jpg --debug
cm find script "python app image-classification onnx"
cm load script "python app image-classification onnx" --yaml
cmr "get python" --version_min=3.8.0 --name=mlperf-experiments
cmr "install python-venv" --version_max=3.10.11 --name=mlperf
cmr "get ml-model stable-diffusion"
cmr "get ml-model sdxl _fp16 _rclone"
cmr "get ml-model huggingface zoo _model-stub.alpindale/Llama-2-13b-ONNX" --model_filename=FP32/LlamaV2_13B_float32.onnx --skip_cache
cmr "get dataset coco _val _2014"
cmr "get dataset openimages" -j
cm show cache
cm show cache "get ml-model stable-diffusion"
cmr "get generic-python-lib _package.onnxruntime" --version_min=1.16.0
cmr "python app image-classification onnx" --input=computer_mouse.jpg
cm rm cache -f
cmr "python app image-classification onnx" --input=computer_mouse.jpg --adr.onnxruntime.version_max=1.16.0
cmr "get cuda" --version_min=12.0.0 --version_max=12.3.1
cmr "python app image-classification onnx _cuda" --input=computer_mouse.jpg
cm gui script "python app image-classification onnx"
cm docker script "python app image-classification onnx" --input=computer_mouse.jpg
cm docker script "python app image-classification onnx" --input=computer_mouse.jpg -j -docker_it
cm docker script "get coco dataset _val _2017" --to=d:\Downloads\COCO-2017-val --store=d:\Downloads --docker_cm_repo=ctuning@mlcommons-ck
cmr "run common mlperf inference" --implementation=nvidia --model=bert-99 --category=datacenter --division=closed
cm find script "run common mlperf inference"
cmr "get generic-python-lib _package.torch" --version=2.1.2
cmr "get generic-python-lib _package.torchvision" --version=0.16.2
cmr "python app image-classification torch" --input=computer_mouse.jpg
cm rm repo mlcommons@ck
cm pull repo --url=https://zenodo.org/records/10787459/files/cm-mlops-repo-20240306.zip
cmr "install llvm prebuilt" --version=17.0.6
cmr "app image corner-detection"
cm run experiment --tags=tuning,experiment,batch_size -- echo --batch_size={{VAR1{range(1,8)}}}
cm replay experiment --tags=tuning,experiment,batch_size
cmr "get conda"
cm pull repo ctuning@cm4research
cmr "reproduce paper micro-2023 victima _install_deps"
cmr "reproduce paper micro-2023 victima _run"
Click to see a few examples of modular containers and GitHub actions with CM commands:
- MLPerf-Bench @ HPCA'24 "Automatically Composing High-Performance and Cost-effective AI Systems with MLCommons' CM and MLPerf": [ slides ]
- ACM REP'23 keynote about MLCommons CM: [ slides ] [ YouTube ]
- ACM TechTalk'21 about automating research projects: [ YouTube ] [ slides ]
- MLPerf inference submitter orientation: [ v4.0 slides ] [ v3.1 slides ]
Collective Mind is being developed by the MLCommons Task Force on Automation and Reproducibility as a collaborative effort with great help from MLCommons members and individual contributors - our goal is to help everyone automate all manual and repetitive tasks to build, run, benchmark and optimize AI systems including downloading artifacts, installing tools, resolving dependencies, running experiments, processing logs, and reproducing results on any software/hardware stack - don't hesitate to get in touch via public Discord server!