Cleaning up modules for MLOps and preparing a new release

CHANGES.txt

@@ -1,7 +1,7 @@
-* v2.5.6.1
+* v2.5.7
   * added 'ck_html_end_note' key to customize CK result dashboard
   * fixed Pareto frontier filter
-  * added "ck filter_2d math.frontier"
+  * added "ck filter_2d math.frontier" for MLPerf inference
 
 * v2.5.6
   * added --j flag to "ck install package" to update CK_HOST_CPU_NUMBER_OF_PROCESSORS env 

README.md

@@ -30,7 +30,7 @@ Windows: [![Windows Build status](https://ci.appveyor.com/api/projects/status/iw
 * [Project website](https://cKnowledge.org)
 * [CK-powered MLPerf™ benchmark automation](https://github.com/ctuning/ck/blob/master/docs/mlperf-automation/README.md)
 * [Community projects to improve and redesign CK](https://github.com/ctuning/ck/blob/master/incubator/README.md)
-* [AI/ML repository with all aggregated CK components](https://github.com/ctuning/ai)
+* [CK automation recipes for portable MLOps](https://github.com/ctuning/ck-mlops)
 * [OctoML's CK-based MLOps/MLPerf repository](https://github.com/octoml/mlops)
 
 ## Overview
@@ -49,7 +49,7 @@ simplifies [MLPerf™ inference benchmark](https://github.com/ctuning/ck/blo
 and supports collaborative, reproducible and reusable ML Systems research:
 
 * [ACM TechTalk](https://www.youtube.com/watch?v=7zpeIVwICa4)
-* [AI/ML/MLPerf™ automation workflows and components from the community](https://github.com/ctuning/ai);
+* [AI/ML/MLPerf™ automation workflows and components from the community](https://github.com/ctuning/ck-mlops);
 * [Reddit discussion about reproducing 150 papers](https://www.reddit.com/r/MachineLearning/comments/ioq8do/n_reproducing_150_research_papers_the_problems);
 * Our reproducibility initiatives: [methodology](https://cTuning.org/ae), [checklist](https://ctuning.org/ae/submission_extra.html), [events](https://cKnowledge.io/events).
 
@@ -135,7 +135,7 @@ Check [CK docs](https://ck.readthedocs.io/en/latest/src/introduction.html) for f
 ### Portable CK workflow (with Docker)
 
 We have prepared m CK containers with ML Systems components:
-* https://github.com/ctuning/ai/tree/main/docker
+* https://github.com/ctuning/ck-mlops/tree/main/docker
 * https://github.com/octoml/mlops/tree/main/docker
 
 You can run them as follows:

ck/kernel.py

@@ -28,7 +28,7 @@
 
 
 # We use 3 digits for the main (released) version and 4th digit for development revision
-__version__ = "2.5.6.1"
+__version__ = "2.5.7"
 # Do not use characters (to detect outdated version)!
 
 # Import packages that are global for the whole kernel

ck/repo/module/advice/module.py

@@ -16,16 +16,10 @@
 # Local settings
 
 hextra='<center>\n'
-hextra+=' [ <a href="http://cKnowledge.org/ai.html">Project website</a> ], '
-hextra+=' [ <a href="http://cKnowledge.org/partners.html">Partners</a> ], '
-hextra+=' [ <a href="http://dividiti.blogspot.fr/2017/02/we-received-test-of-time-award-for-our.html">CGO\'17 test of time award for our interdisiplinary R&D</a> ], '
-hextra+=' [ <a href="http://cKnowledge.org/ai">Community-driven AI R&D powered by CK</a> ], '
-hextra+=' [ <a href="https://github.com/dividiti/ck-caffe">CK-Caffe</a> ], '
-hextra+=' [ <a href="https://github.com/ctuning/ck-tensorflow">CK-TensorFlow</a> ], '
-hextra+=' [ <a href="https://en.wikipedia.org/wiki/Collective_Knowledge_(software)">Wikipedia</a>, \n'
-hextra+='<a href="https://www.researchgate.net/publication/304010295_Collective_Knowledge_Towards_RD_Sustainability">paper 1</a>, \n'
-hextra+='<a href="https://arxiv.org/abs/1506.06256">Paper 2</a>, \n'
-hextra+='<a href="https://www.youtube.com/watch?v=Q94yWxXUMP0">YouTube CK intro</a> ] \n'
+hextra+=' [ <a href="https://cKnowledge.org">Project website</a> ], '
+hextra+=' [ <a href="https://cKnowledge.org/partners.html">Partners</a> ], '
+hextra+=' [ <a href="https://github.com/ctuning/ck-mlops">CK automation recipes for MLOps</a> ], '
+hextra+=' [ <a href="https://en.wikipedia.org/wiki/Collective_Knowledge_(software)">Wikipedia</a> ] \n'
 hextra+='</center>\n'
 hextra+='\n'
 

ck/repo/module/env/module.py

@@ -392,7 +392,7 @@ def env_set(i):
              ck.out('')
              ck.out(' This is a possible bug - please report here:')
              ck.out('   * https://github.com/ctuning/ck/issues')
-             ck.out('   * https://github.com/ctuning/ai/issues')
+             ck.out('   * https://github.com/ctuning/ck-mlops/issues')
              ck.out('')
 
           return {'return':33, 'error':'current host or target OS ('+str(setup)+' is not matching the one in software env '+duoa}

ck/repo/module/experiment.bench.caffe2/module.py

@@ -24,25 +24,12 @@
 form_name='wa_web_form'
 onchange='document.'+form_name+'.submit();'
 
-#hextra='<i><center>\n'
-#hextra+='This is an on-going long-term project. Please check our vision [ '
-#hextra+='<a href="http://doi.acm.org/10.1145/2909437.2909449">IWOCL\'16</a>, \n'
-#hextra+='<a href="http://arxiv.org/abs/1506.06256">CPC\'15</a>, \n'
-#hextra+='<a href="https://www.youtube.com/watch?v=Q94yWxXUMP0">YouTube</a>, \n'
-#hextra+='<a href="http://ctuning.org/cm/wiki/index.php?title=CM:data:45741e3fbcf4024b:1db78910464c9d05">wiki</a> ] '
-#hextra+=' and <a href="https://github.com/ctuning/ck-tensorflow">CK-TensorFlow GitHub repo</a> for more details!'
-#hextra+='</center></i>\n'
-#hextra+='<br>\n'
-
 hextra='<i><center>\n'
-hextra+=' [ <a href="http://cKnowledge.org/ai">Collaborative unification of AI</a> ], '
-hextra+=' [ <a href="https://github.com/ctuning/ck-caffe2">CK-Caffe2</a> / <a href="https://github.com/dividiti/ck-caffe">CK-Caffe</a> ], '
-hextra+=' [ <a href="https://github.com/ctuning/ck-tensorflow">CK-Tensorflow</a> ], '
+hextra+=' [ <a href="https://cKnowledge.org">Project website</a> ], '
+hextra+=' [ <a href="https://github.com/ctuning/ck-mlops">CK automation recipes for portable MLOps</a> ], '
 hextra+=' [ <a href="http://cKnowledge.org/android-apps.html">Android app</a> ], '
 hextra+=' [ <a href="https://en.wikipedia.org/wiki/Collective_Knowledge_(software)">CK intro</a>, \n'
-hextra+='<a href="https://www.researchgate.net/publication/304010295_Collective_Knowledge_Towards_RD_Sustainability">vision</a> and \n'
-hextra+='<a href="https://arxiv.org/abs/1506.06256">crowd-tuning</a>; \n'
-hextra+='<a href="https://www.youtube.com/watch?v=Q94yWxXUMP0">YouTube lecture</a> ] \n'
+hextra+='<a href="https://arxiv.org/abs/1506.06256">crowd-tuning</a> ] \n'
 hextra+='</center></i>\n'
 hextra+='<br>\n'
 

ck/repo/module/experiment.bench.dnn.mobile/module.py

@@ -25,18 +25,6 @@
 onchange='document.'+form_name+'.submit();'
 
 hextra=''
-#hextra+='<i><center>\n'
-#hextra+=' [ <a href="http://cKnowledge.org/ai">Community-driven AI R&D powered by CK</a> ], '
-#hextra+=' [ <i><a href="http://dividiti.blogspot.fr/2017/02/we-received-test-of-time-award-for-our.html">CGO\'17 test of time award for our interdisiplinary R&D</a></i> ], '
-#hextra+=' [ <b><a href="http://cKnowledge.org/android-apps.html">Android app to crowd-optimize DNN engines and models</a></b> ], '
-#hextra+=' [ <a href="https://github.com/ctuning/ck-caffe2">CK-Caffe2 GitHub</a> / <a href="https://github.com/dividiti/ck-caffe">CK-Caffe GitHub</a> ], '
-#hextra+=' [ <a href="https://github.com/ctuning/ck-tensorflow">CK-TensorFlow GitHub</a> ], '
-#hextra+=' [ <a href="https://en.wikipedia.org/wiki/Collective_Knowledge_(software)">Wikipedia</a>, \n'
-#hextra+='<a href="https://www.researchgate.net/publication/304010295_Collective_Knowledge_Towards_RD_Sustainability">paper 1</a>, \n'
-#hextra+='<a href="https://arxiv.org/abs/1506.06256">Paper 2</a>, \n'
-#hextra+='<a href="https://www.youtube.com/watch?v=Q94yWxXUMP0">YouTube CK intro</a> ] \n'
-#hextra+='</center></i>\n'
-#hextra+='<br>\n'
 
 selector=[{'name':'Scenario', 'key':'crowd_uid', 'module_uoa':'65477d547a49dd2c', 'module_key':'##dict#title'},
           {'name':'DNN engine', 'key':'engine'},

ck/repo/module/experiment.bench.dnn/module.py

@@ -24,25 +24,10 @@
 form_name='wa_web_form'
 onchange='document.'+form_name+'.submit();'
 
-#hextra='<i><center>\n'
-#hextra+='This is an on-going long-term project. Please check our vision [ '
-#hextra+='<a href="http://doi.acm.org/10.1145/2909437.2909449">IWOCL\'16</a>, \n'
-#hextra+='<a href="http://arxiv.org/abs/1506.06256">CPC\'15</a>, \n'
-#hextra+='<a href="https://www.youtube.com/watch?v=Q94yWxXUMP0">YouTube</a>, \n'
-#hextra+='<a href="http://ctuning.org/cm/wiki/index.php?title=CM:data:45741e3fbcf4024b:1db78910464c9d05">wiki</a> ] '
-#hextra+=' and <a href="https://github.com/dividiti/ck-caffe">CK-Caffe GitHub repo</a> for more details!'
-#hextra+='</center></i>\n'
-#hextra+='<br>\n'
-
 hextra='<i><center>\n'
-hextra+=' [ <a href="http://cKnowledge.org/ai">Community-driven AI R&D powered by CK</a> ], '
-hextra+=' [ <a href="https://github.com/ctuning/ck-caffe2">CK-Caffe2</a> / <a href="https://github.com/dividiti/ck-caffe">CK-Caffe</a> ], '
-hextra+=' [ <a href="https://github.com/ctuning/ck-tensorflow">CK-TensorFlow</a> ], '
-hextra+=' [ <a href="https://en.wikipedia.org/wiki/Collective_Knowledge_(software)">Wikipedia</a>, \n'
-hextra+='<a href="https://www.researchgate.net/publication/304010295_Collective_Knowledge_Towards_RD_Sustainability">paper 1</a>, \n'
-hextra+='<a href="https://arxiv.org/abs/1506.06256">Paper 2</a>, \n'
-hextra+='<a href="https://www.youtube.com/watch?v=Q94yWxXUMP0">YouTube CK intro</a> ], \n'
-hextra+=' [ <a href="http://dividiti.blogspot.fr/2017/02/we-received-test-of-time-award-for-our.html">CGO\'17 test of time award for our interdisiplinary R&D</a> ]'
+hextra+=' [ <a href="http://cKnowledge.org">CK project website</a> ], '
+hextra+=' [ <a href="https://github.com/ctuning/ck-mlops">CK automation recipes for portable MLOps</a> ], '
+hextra+=' [ <a href="https://en.wikipedia.org/wiki/Collective_Knowledge_(software)">Wikipedia</a> ] \n'
 hextra+='</center></i>\n'
 hextra+='<br>\n'
 

ck/repo/module/experiment.bench.tensorflow/module.py

@@ -24,25 +24,10 @@
 form_name='wa_web_form'
 onchange='document.'+form_name+'.submit();'
 
-#hextra='<i><center>\n'
-#hextra+='This is an on-going long-term project. Please check our vision [ '
-#hextra+='<a href="http://doi.acm.org/10.1145/2909437.2909449">IWOCL\'16</a>, \n'
-#hextra+='<a href="http://arxiv.org/abs/1506.06256">CPC\'15</a>, \n'
-#hextra+='<a href="https://www.youtube.com/watch?v=Q94yWxXUMP0">YouTube</a>, \n'
-#hextra+='<a href="http://ctuning.org/cm/wiki/index.php?title=CM:data:45741e3fbcf4024b:1db78910464c9d05">wiki</a> ] '
-#hextra+=' and <a href="https://github.com/ctuning/ck-tensorflow">CK-TensorFlow GitHub repo</a> for more details!'
-#hextra+='</center></i>\n'
-#hextra+='<br>\n'
-
 hextra='<i><center>\n'
-hextra+=' [ <a href="http://cKnowledge.org/ai">Collaborative unification of AI</a> ], '
-hextra+=' [ <a href="https://github.com/ctuning/ck-caffe2">CK-Caffe2</a> / <a href="https://github.com/dividiti/ck-caffe">CK-Caffe</a> ], '
-hextra+=' [ <a href="https://github.com/ctuning/ck-tensorflow">CK-Tensorflow</a> ], '
-hextra+=' [ <a href="http://cKnowledge.org/android-apps.html">Android app</a> ], '
-hextra+=' [ <a href="https://en.wikipedia.org/wiki/Collective_Knowledge_(software)">CK intro</a>, \n'
-hextra+='<a href="https://www.researchgate.net/publication/304010295_Collective_Knowledge_Towards_RD_Sustainability">vision</a> and \n'
-hextra+='<a href="https://arxiv.org/abs/1506.06256">crowd-tuning</a>; \n'
-hextra+='<a href="https://www.youtube.com/watch?v=Q94yWxXUMP0">YouTube lecture</a> ] \n'
+hextra+=' [ <a href="https://cKnowledge.org">CK project website</a> ], '
+hextra+=' [ <a href="https://github.com/ctuning/ck-mlops">CK automation recipes for portable MLOps</a> ], '
+hextra+=' [ <a href="http://cKnowledge.org/android-apps.html">Android app</a> ]\n '
 hextra+='</center></i>\n'
 hextra+='<br>\n'
 

ck/repo/module/model.image.classification/module.py

@@ -13,14 +13,9 @@
 
 # Local settings
 hextra='<i><center>\n'
-hextra+=' [ <a href="http://cKnowledge.org/ai">Community-driven AI R&D powered by CK</a> ], '
-hextra+=' [ <a href="https://github.com/dividiti/ck-caffe">CK-Caffe</a> ], '
-hextra+=' [ <a href="https://github.com/ctuning/ck-caffe2">CK-Caffe2</a> ], '
-hextra+=' [ <a href="https://github.com/ctuning/ck-tensorflow">CK-TensorFlow</a> ], '
-hextra+=' [ <a href="https://en.wikipedia.org/wiki/Collective_Knowledge_(software)">Wikipedia</a>, \n'
-hextra+='<a href="https://www.researchgate.net/publication/304010295_Collective_Knowledge_Towards_RD_Sustainability">paper 1</a>, \n'
-hextra+='<a href="https://arxiv.org/abs/1506.06256">Paper 2</a>, \n'
-hextra+='<a href="https://www.youtube.com/watch?v=Q94yWxXUMP0">YouTube CK intro</a> ] \n'
+hextra+=' [ <a href="https://cKnowledge.org">CK project website</a> ], '
+hextra+=' [ <a href="https://github.com/ctuning/ck-mlops">CK automation recipes for portable MLOps</a> ], '
+hextra+=' [ <a href="https://en.wikipedia.org/wiki/Collective_Knowledge_(software)">Wikipedia</a> ]  \n'
 hextra+='</center></i>\n'
 hextra+='<br>\n'
 

ck/repo/module/package/module.py

@@ -2872,7 +2872,7 @@ def print_warning(i):
 #       x2='https://groups.google.com/forum/#!forum/collective-knowledge'
 #       x3='https://github.com/ctuning/ck/issues'
        x2='https://github.com/ctuning/ck/issues'
-       x3='https://github.com/ctuning/ai/issues'
+       x3='https://github.com/ctuning/ck-mlops/issues'
 #       if url2!='':
 #          x1='the authors'
 #          x2=url2

ck/repo/module/program.optimization/module.py

@@ -3132,21 +3132,9 @@ def links(i):
     h+='  <a href="http://cknowledge.org/repo/web.php?action=index&module_uoa=wfe&native_action=show&native_module_uoa=platform.npu">NPU</a> ] \n'
     h+='[ <a href="https://github.com/ctuning/ck/wiki/Crowdsourcing-optimization">How to participate</a> ] \n'
     h+='[ Motivation (<a href="https://www.slideshare.net/GrigoriFursin/adapting-to-a-cambrian-aiswhw-explosion-with-open-codesign-competitions-and-collective-knowledge">PPT</a>) (<a href="https://portalparts.acm.org/3230000/3229762/fm/frontmatter.pdf">PDF</a>) ] \n'
-    h+='[ Papers <a href="https://arxiv.org/abs/1801.08024">1</a> , <a href="https://arxiv.org/abs/1506.06256">2</a> , <a href="https://www.researchgate.net/publication/304010295_Collective_Knowledge_Towards_RD_Sustainability">3</a>] \n'
-    h+='[ <a href="http://cKnowledge.org/android-apps.html"><b>Android app</b></a> ] \n'
+    h+='[ <a href="https://cKnowledge.org/android-apps.html"><b>Android app</b></a> ] \n'
     h+='[ <b><a href="http://cknowledge.org/repo/web.php?wcid=42b9a1221eb50259:collective_training_set">Collective training set</a></b> ] \n'
-    h+='[ <b><a href="http://cKnowledge.org/ai">Unified AI</a></b> ] \n'
-
-#    h+='[ <a href="https://github.com/ctuning/ck"><b>open research SDK</b></a> ], \n'
-#    h+='[ <b>Android apps to crowdsource experiments:</b> <a href="http://cKnowledge.org/android-apps.html">small kernels</a>, <a href="https://play.google.com/store/apps/details?id=openscience.crowdsource.video.experiments">apps (DNN)</a>) ], \n'
-#    h+='[ A few papers: <a href="http://arxiv.org/abs/1506.06256">CPC\'15</a>, \n'
-#    h+='  <a href="https://www.researchgate.net/publication/304010295_Collective_Knowledge_Towards_RD_Sustainability">DATE\'16</a>, \n'
-#    h+='  <a href="http://arxiv.org/abs/1406.4020">TRUST@PLDI\'14</a>, <a href="http://cknowledge.org/interactive-report">interactive</a>,\n'
-#    h+='  <a href="https://www.youtube.com/watch?v=Q94yWxXUMP0">YouTube</a>\n'
-#    h+='  ], \n'
-#    h+='[ <a href="http://cTuning.org/ae">Our reproducible initiative for ACM conferences</a> ], \n'
-#    h+='[ <a href="http://dividiti.blogspot.fr/2017/02/we-received-test-of-time-award-for-our.html">CGO\'17 test of time award for our interdisiplinary R&D</a> ], '
-#    h+='[ <b><a href="http://cknowledge.org/ai/ck-api-demo">Open and unified CK API for AI</a></b> ] '
+    h+='[ <b><a href="http://cKnowledge.org">CK project website</a></b> ] \n'
 
     h+='</center>\n'
     h+='</div>\n'

ck/repo/module/program/module.py

@@ -7766,7 +7766,7 @@ def print_warning(i):
 #       x2='https://groups.google.com/forum/#!forum/collective-knowledge'
 #       x3='https://github.com/ctuning/ck/issues'
        x2='https://github.com/ctuning/ck/issues'
-       x3='https://github.com/ctuning/ai/issues'
+       x3='https://github.com/ctuning/ck-mlops/issues'
 #       if url2!='':
 #          x1='the authors'
 #          x2=url2

docs/mlperf-automation/README.md

@@ -6,8 +6,8 @@ We also want to develop an open database for benchmarking results and provenance
 compatible with [FAIR principles](https://www.go-fair.org/fair-principles).
 
 As a starting point, we will use the open-source and technology-neutral [CK framework](https://github.com/ctuning/ck) 
-with a [collection](https://github.com/ctuning/ai) of [reusable automation recipes](https://github.com/ctuning/ai/tree/main/program) 
-and [plug&play packages](https://github.com/ctuning/ai/tree/main/package) 
+with a [collection](https://github.com/ctuning/ck-mlops) of [reusable automation recipes](https://github.com/ctuning/ck-mlops/tree/main/program) 
+and [plug&play packages](https://github.com/ctuning/ck-mlops/tree/main/package) 
 for ML systems. CK framework was already successfully used by several MLCommons members 
 to automate their MLPerf inference submissions and we want to build upon their experience. 
 
@@ -16,7 +16,7 @@ open-source tools from the MLCommons™ Best Practices WorkGroup ([MLCube&tr
 
 A few examples: 
 * [MLPerf™ object detection workflow](https://github.com/ctuning/ck/blob/master/docs/mlperf-automation/tasks/task-object-detection.md)
-* [Docker image for MLPerf™ with OpenVINO]( https://github.com/ctuning/ai/tree/main/docker/mlperf-inference-v0.7.openvino )
+* [Docker image for MLPerf™ with OpenVINO]( https://github.com/ctuning/ck-mlops/tree/main/docker/mlperf-inference-v0.7.openvino )
 * [Jupyter notebook for ML DSE](https://nbviewer.jupyter.org/urls/dl.dropbox.com/s/f28u9epifr0nn09/ck-dse-demo-object-detection.ipynb)
 * [Webcam test of the MLPerf object detection model with TFLite](https://cknowledge.io/solution/demo-obj-detection-coco-tf-cpu-webcam-linux-azure#test)
 * [Public scoreboard with MLPerf DSE](https://cknowledge.io/result/crowd-benchmarking-mlperf-inference-classification-mobilenets-all)

docs/mlperf-automation/components/README.md

@@ -3,7 +3,7 @@
 # CK components for ML Systems (automation recipes)
 
 Over the past few years we've collected many CK components
-and automation recipes for ML Systems in [this public repo](https://github.com/ctuning/ai)
+and automation recipes for ML Systems in [this public repo](https://github.com/ctuning/ck-mlops)
 (ML models, frameworks, tools, data sets, portable workflows, autotuning scripts, etc).
 
 We now work with the community and MLCommons to test and unify these components.

docs/mlperf-automation/inference/containers.md

@@ -4,7 +4,7 @@ We would like to prepare a set of containers to run MLPerf inference benchmark o
 
 Ongoing efforts:
 * https://github.com/octoml/mlops/tree/main/docker
-* https://github.com/ctuning/ai/tree/main/docker
+* https://github.com/ctuning/ck-mlops/tree/main/docker
 
 
 ## Adaptive CK containers for MLPerf inference

docs/mlperf-automation/inference/workflow.md

@@ -33,7 +33,7 @@ We plan to add module:mlperf.bench.inference with the following flags:
 
 ```
 
-This high-level workflow will install relevant CK packages ([ctuning@ai](https://github.com/ctuning/ai/tree/main/package) / [octoml@mlops](https://github.com/octoml/mlops/tree/main/package))
+This high-level workflow will install relevant CK packages ([ctuning@ai](https://github.com/ctuning/ck-mlops/tree/main/package) / [octoml@mlops](https://github.com/octoml/mlops/tree/main/package))
 and run low-level [CK program workflows for MLPerf](https://github.com/octoml/mlops/tree/main/program).