Skip to content

Commit

Permalink
updating docs (#1236)
Browse files Browse the repository at this point in the history
  • Loading branch information
@ctuning-admin
ctuning-admin authored Apr 29, 2024
2 parents af186af + 94a9396 commit 2d08dbd
Show file tree
Hide file tree
Showing 6 changed files with 66 additions and 56 deletions.
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Ensure that cla-bot and other checks pass for your Pull requests.

* Grigori Fursin (cTuning.org and cKnowledge.org)

## CM contributors in alphabetical order (suggestions and/or scripts)
## CM contributors in alphabetical order (suggestions, feedback, scripts)

* Resmi Arjun
* Alice Cheng (Nvidia)
Expand Down
2 changes: 1 addition & 1 deletion COPYRIGHT.txt
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
Copyright (c) 2021-2024 MLCommons

The cTuning foundation and OctoML donated this project to MLCommons to benefit everyone.
The cTuning foundation donated this project to MLCommons to benefit everyone.

Copyright (c) 2014-2021 cTuning foundation
22 changes: 9 additions & 13 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@ across diverse and continuously changing models, data, software and hardware.
CK consists of several ongoing sub-projects:

* [Collective Mind framework (CM)](cm) (*~1MB*) - a very light-weight Python-based framework with minimal dependencies
to help the community implement, share and reuse cross-platform automation recipes to
to help users implement, share and reuse cross-platform automation recipes to
build, benchmark and optimize applications on any platform
with any software and hardware. Very simply, it extends the cmake concept with
reusable automation recipes and workflows written in plain Python or native OS scripts,
with any software and hardware. CM attempts to extends the `cmake` concept
with reusable automation recipes and workflows written in plain Python or native OS scripts,
accessible via a human readable interface with simple tags,
and shareable in public and private repositories in a decentralized way.
Furthermore, in comparison with cmake, these automation recipes can not only detect missing code
Expand Down Expand Up @@ -62,31 +62,27 @@ CK consists of several ongoing sub-projects:

### Motivation behind CK and CM projects

* ACM REP'23 keynote about MLCommons CM: [ [slides](https://doi.org/10.5281/zenodo.8105339) ]
* ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ]
* ACM TechTalk'21 about automating research projects: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ]

### Documentation

**We plan to rewrite and simplify the CM documentation and tutorials based on user feedback in Q2 2024 - please stay tuned for more details**.

* [News](docs/news.md)
* [Getting Started Guide and FAQ](docs/getting-started.md)
* [CM Getting Started Guide and FAQ](docs/getting-started.md)
* [Common CM interface to run MLPerf inference benchmarks](docs/mlperf/inference)
* [Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23](docs/tutorials/common-interface-to-reproduce-research-projects.md)
* [CM automation recipes for MLOps and DevOps](cm-mlops/script)
* [Other CM tutorials](docs/tutorials)
* [Full documentation](docs/README.md)
* [CM development tasks](docs/taskforce.md#current-tasks)
* [CM and CK history](docs/history.md)


### Get in touch

Collective Mind workflow automation framework and Collective Knowledge Playground are being developed
by the [MLCommons Task Force on Automation and Reproducibility](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md)
as a community effort. Volunteers are very welcome to help with this community project!

### Acknowledgments

CK and CM are community projects based on the feedback from our users and MLCommons members.
We would like to thank all [collaborators and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)
This open-source technology is being developed by the [MLCommons Task Force on Automation and Reproducibility](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md)
as a community effort based on user feedback.
We would like to thank all our [volunteers, collaborators and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)
for their support, fruitful discussions, and useful feedback!
4 changes: 3 additions & 1 deletion docs/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,14 @@

Collective Mind (CM) is a lightweight, non-intrusive and technology-agnostic workflow automation framework
being developed by the [MLCommons Task Force on Automation and Reproducibility](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md)
based on the feedback from the [the community, MLCommons members and individual contributors](CONTRIBUTING.md).
based on the feedback from the [the community, MLCommons members and individual contributors](../CONTRIBUTING.md).

The goal is to provide a common, simple and human-readable interface to help users encode their knowledge
about how to build, run and customize diverse AI/ML apps, benchmarks and research projects across
continuously changing models, datasets, software and hardware from different vendors in a unified and automated way.

You can find on-going development tasks [here](https://github.com/mlcommons/ck/blob/dev/docs/taskforce.md#current-tasks).

* [Getting Started Guide and FAQ](getting-started.md)
* [Introduction](introduction-cm.md)
* [CM installation and customization](installation.md)
Expand Down
20 changes: 9 additions & 11 deletions docs/history.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,14 @@ The archive of this discontinued framework is available in the [/ck directory](.

# MLCommons Collective Mind (CM)

Grigori donated the CK technology to [MLCommons](https://mlcommons.org) in 2022 to benefit everyone and help establish
the [MLCommons Task Force on Automation and Reproducibility](taskforce.md) co-led with [Arjun Suresh](https://www.linkedin.com/in/arjunsuresh).
Grigori donated the CK technology to [MLCommons](https://mlcommons.org) in 2022 to benefit everyone,
prototyped a new version of CK called [Collective Mind (CM)](https://github.com/mlcommons/ck/tree/master/cm)
with [portable and reusable automation recipes for universal benchmarking and optimization of applications
and systems (CM scripts)](https://access.cknowledge.org/playground/?action=scripts),
and helped establish the [MLCommons Task Force on Automation and Reproducibility](taskforce.md)
co-led with Arjun Suresh.

This collaborative engineering effort resulted in the new version of CK called Collective Mind (CM)
with [simple, portable and ready-to-use automation recipes](list_of_scripts.md) for MLOps and DevOps
developed by scratch with great help, feedback and contributions
from [the community](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md).
We continue extending CM to support different MLCommons projects to modularize and unify benchmarking
of ML/AI systems as a collaborative engineering effort based on [user feedback](../CONTRIBUTING.md).

CM is now officially supported, developed and maintained by [MLCommons.org](https://mlcommons.org)
with the help from [cKnowledge.org](https://cKnowledge.org) and [cTuning.org](https://cTuning.org).

Please go to this [GitHub page](https://github.com/mlcommons/ck) to learn more about this project
and check [the slides from ACM REP'23 keynote about CM](https://doi.org/10.5281/zenodo.8105339).
You can learn more about the CM concept and motivation from the [keynote at ACM REP'23](https://doi.org/10.5281/zenodo.8105339).
72 changes: 43 additions & 29 deletions docs/taskforce.md
View file
Original file line number Diff line number Diff line change
@@ -1,22 +1,19 @@
[ [Back to index](README.md) ]

# MLCommons Task force on Automation and Reproducibility

***Announcement: we are peparing new tasks for Q3-Q4 2024 - please stay tuned
or get in touch via [email](mailto:gfursin@cknowledge.org) for more details!***
# MLCommons Task Force on Automation and Reproducibility

## Mission

* Develop [reusable automation recipes and workflows](https://access.cknowledge.org/playground/?action=scripts)
with [a common and human-friendly interface (Collective Mind aka CM)]( https://github.com/mlcommons/ck/tree/master/cm )
to support MLCommons projects and help everyone assemble, run, reproduce, customize and optimize ML(Perf) benchmarks
* Extend [MLCommons CM workflow automation framework](https://github.com/mlcommons/ck) and
[reusable automation recipes (CM scripts)](https://access.cknowledge.org/playground/?action=scripts)
to automate MLCommons projects and make it easier to assemble, run, reproduce, customize and optimize ML(Perf) benchmarks
in a unified and automated way across diverse models, data sets, software and hardware from different vendors.
* Gradually extend a unified MLCommons CM interface to automate [all MLPerf inference submissions](https://github.com/mlcommons/ck/issues/1052) starting from v3.1.
* Continuously encode MLPerf rules and best practices in the [CM automation recipes and workflows for MLPerf](https://github.com/mlcommons/cm4mlops/tree/main/script)
to reduce the burden for submitters to go through numerous README files
and track all the latest changes and updates.
* Extend [CM workflows](https://github.com/mlcommons/cm4mlops)
to automate and reproduce MLPerf inference submissions from different vendors starting from v3.1.
* Encode MLPerf rules and best practices in the [CM automation recipes and workflows for MLPerf](https://github.com/mlcommons/cm4mlops/tree/main/script)
to help MLPerf submitters avoid going through many README files and track all the latest MLPerf changes and updates.

## Chairs and Tech Leads
## Chairs

* [Grigori Fursin](https://cKnowledge.org/gfursin) (CM project coordinator)
* [Arjun Suresh](https://www.linkedin.com/in/arjunsuresh)
Expand All @@ -25,42 +22,54 @@

Since we already participate in many weekly conf-calls to support various MLCommons initiatives,
we decided not to have yet another separate conf-call for our TF in 2024.

Instead, we discuss our progress on MLPerf automation and reproducibility at existing MLCommons conf-calls
where CM is used:
* [MLPerf inference WG](https://mlcommons.org/working-groups/benchmarks/inference): weekly conf-calls on Tuesday at 8:30am PST
* [MLPerf automotive WG](https://mlcommons.org/working-groups/benchmarks/automotive)
* Automotive TF weekly conf-calls on Wednesday at 8am PST
* Automotive implementation bi-weekly conf-calls on Friday at 9:30am PST
* [Croissant TF](https://github.com/mlcommons/croissant): weekly at 8am PST

You can participate in our discussions via [public Discord server](https://discord.gg/JjWNWXKxwT).
You can also join our public Discord server [here](https://discord.gg/JjWNWXKxwT).


## Current projects

## Deliverables (2024)
* Continue improving CM to support different MLCommons projects for universal benchmarking and optimization across different platforms.

* Collaborate with chip vendors and MLPerf inference submitters to add
their implementations to CM and automate their submissions.
* Extend CM workflows to reproduce MLPerf inference v4.0 submissions (Intel, Nvidia, Qualcomm, Google, Red Hat, etc) via a unified interface.

* Prepare tutorial for MLPerf inference v4.1 submissions via CM.

* Discuss how to introduce the [CM automation badge]( https://github.com/mlcommons/ck/blob/master/docs/artifact-evaluation/reviewing.md )
to MLPerf inference v4.1 submission similar to ACM/IEEE/NeurIPS reproducibility badges to make it easier for
all submitters to re-run and reproduce each others’ results before the publication date.

* Develop a more universal Python and C++ wrapper for the MLPerf loadgen
with the CM automation to support different models, data sets, software
and hardware: [Python prototype](https://github.com/mlcommons/cm4mlops/tree/main/script/app-loadgen-generic-python);
[C++ prototype](https://github.com/mlcommons/cm4mlops/tree/main/script/app-mlperf-inference-mlcommons-cpp).

* Discuss how to introduce the [CM automation badge](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference-mlcommons-cpp)
to MLPerf inference v4.1 submission similar to ACM/IEEE/NeurIPS reproducibility badges to make it easier for
all submitters to re-run and reproduce each others’ results before the publication date.

* Collaborate with system vendors and cloud providers to help them benchmark
their platforms using the best available MLPerf inference implementation.

* Collaborate with other MLPerf working groups to modularize their
benchmarks using [CM automation recipes](https://access.cknowledge.org/playground/?action=scripts).
* Collaborate with other MLCommons working groups to autoamte, modularize and unify
their benchmarks using [CM automation recipes](https://access.cknowledge.org/playground/?action=scripts).

* Use MLCommons CM to modularize and automate the upcoming [automotive benchmark](https://mlcommons.org/working-groups/benchmarks/automotive/).
* Use CM to modularize and automate the upcoming [automotive benchmark](https://mlcommons.org/working-groups/benchmarks/automotive/).

* Use [MLCommons Croissant](https://mlcommons.org/working-groups/data/croissant/)
to unify [MLPerf datasets](https://access.cknowledge.org/playground/?action=scripts).


## Current tasks

* Improving CM workflow automation framework: [GitHub ticket](https://github.com/mlcommons/ck/issues/1229)
* Updating/refactoring CM docs (framework and MLPef workflows): [GitHub ticket](https://github.com/mlcommons/ck/issues/1220)
* Improving CM scripts to support MLPerf: [GitHub ticket](https://github.com/mlcommons/cm4mlops/issues/21)
* Adding profiling and performance analysis during benchmarking: [GitHub ticket](https://github.com/mlcommons/cm4mlops/issues/23)
* Improving universal build and run scripts to support cross-platform compilation: [GitHub ticket](https://github.com/mlcommons/cm4mlops/issues/24)
* Automate ABTF benchmarking via CM: [GitHub ticket](https://github.com/mlcommons/cm4abtf/issues/6)

## Completed deliverables

Expand Down Expand Up @@ -95,13 +104,18 @@ You can participate in our discussions via [public Discord server](https://disco

## Resources

* [GitHub project](https://github.com/mlcommons/ck)
* [Getting Started Guide](https://github.com/mlcommons/ck/blob/master/docs/getting-started.md)
* [CM GitHub project](https://github.com/mlcommons/ck)
* [CM concept (keynote at ACM REP'23)]( https://doi.org/10.5281/zenodo.8105339 )
* [CM Getting Started Guide](https://github.com/mlcommons/ck/blob/master/docs/getting-started.md)
* [CM-MLPerf commands](https://github.com/mlcommons/ck/tree/master/docs/mlperf)
* [CM-MLPerf GUI](https://access.cknowledge.org/playground/?action=howtorun)
* [Invited talk at MLPerf-Bench @ HPCA'24 about Automatically Composing High-Performance and Cost-effective AI Systems with MLCommons' CM and MLPerf](https://doi.org/10.5281/zenodo.10786893)
* [Invited keynote about CM framework at ACM REP'23]( https://doi.org/10.5281/zenodo.8105339 )
* [ACM artifact review and badging methodology](https://www.acm.org/publications/policies/artifact-review-and-badging-current)
* [Artifact Evaluation at ML and systems conferences](https://cTuning.org/ae)
* [Terminology (ACM/NISO): Repeatability, Reproducibility and Replicability](artifact-evaluation/faq.md#what-is-the-difference-between-repeatability-reproducibility-and-replicability)
* [ACM TechTalk about reproducing 150+ research papers and validating them in the real world](https://www.youtube.com/watch?v=7zpeIVwICa4)
* [CM motivation (ACM TechTalk about reproducing 150+ research papers and validating them in the real world)](https://www.youtube.com/watch?v=7zpeIVwICa4)

## Acknowledgments

This open-source technology is being developed as a community effort based on user feedback.
We would like to thank all our [volunteers, collaborators and contributors](../CONTRIBUTING.md)
for their support, fruitful discussions, and useful feedback!

0 comments on commit 2d08dbd

Please sign in to comment.