NNI 2021 June~July Iteration Planning

[kvartet]

This is the plan for the iteration in June~July Iteration, it's a 6 weeks iteration.

Release Plan for v2.4

• Release manager: @QuanluZhang
• Feature freeze date: July 14
• Code freeze and demo date: July 21 23 26
• Branch cut and next release planning date: July 28 August 2
• Release date: August 2 4 11

Retiarii & NAS

• P0 - NAS visualization
  • trial detail/content re-format @ultmaster 6.29 ([Retiarii] Visualization #3878)
  • integrate Netron @98may @ultmaster
• P0 - Add examples for NAS-Bench-series @ultmaster
  • 101 6.25 [Retiarii] NAS-Bench-101 #3871
  • 201 [Retiarii] NAS-Bench-201 #3920
  • P2 - 301
• P2 - Evaluate existing strategies (multi-trial) on NAS benchmarks @ultmaster
• P0 - Support one system-level optimization: CSE optimization to deduplicate computation @hzhua pr [Retiarii] cross-graph optimization: device placement and input deduplication #3202
• P0 - Improve training service to support CSE optimization @hzhua extend reusable training service to support placement constraint #3897
• P2 - TensorFlow support: graph converter and support inline mutation API @liuzhe-lz
  • add raw framework
  • add example 7.12
• P0 - Migrate FBNet from NAS v1.0 to Retiarii framework @ali
• P2 - AutoDropout @skeletondyh 7.16
• P0 - Searching for activation functions. @QuanluZhang 6.24 6.28 ([Retiarii] support hypermodule: autoactivation #3868 )
  • P0 - (discussion) support autoactivation on base execution engine 6.29
• P1 - Support PyTorch 1.8/1.9 @QuanluZhang 7.14 [retiarii] support torch 1.8 and 1.9 #3937
• P0 - Deal with retiarii experiment exit failure @QuanluZhang 7.1 7.2 fix multi-thread event handling issue #3899
• P1 - survey serialize approaches @ultmaster 7.15 7.19
• P0 - support nn-meter in multi-trial nas @ultmaster @jianyu [DO NOT SQUASH] Support nn-Meter in Retiarii framework #3938

Model Compression

• P0 - Model Speedup Refactor @zheng-ningxin pr Model Speedup Refactor #3462
• P1 - Add more examples for model speedup @zheng-ningxin Add several speedup examples #3880
• P0 - Add test for TensorRT @linbinskn 6.25 @J-shang
• P0 - Global sort for taylor @linbinskn 7.2 [Model Compression] Add global sort for taylor pruner #3896
  • support constraint
• P0 - support iterative pruning @J-shang 6.18 review meeting (done) / 6.23 sample / 6.25 7.14
  • P0 - implement model compression 2.0, include 8 maskers (basic pruners) 7.9 Compression v2 Stage 1 #3917
  • P0 - improve pruners to more faithfully reproduce the papers @J-shang
  • P2 - Refactor compress() in IterativePruner, unify support AGP, ADMM. (iterative pruner follow-up  #3669 )
  • P2 - Add more criteria(strategy, i.e., acc drop toleronce) (discussion needed)
• P0 - fix exclude not support in partial compressor configlist @J-shang pr ready add exclude config validate in compressor #3815
• P0 - review meeting of model compression @QuanluZhang 6.22
• P1 - Add one compression algorithm for Bert/Transformer @xiaowu0162
  • review meeting 6.23
  • first version implementation 7.1 7.9 Transformer Head Pruner #3884

Training service & NNI manager && nnictl

• P0 - file name validation, from error to warning and improve warning message @liuzhe-lz (Change file name validation to warning #3843)
• P0 - JuypterLab support - JupyterLab support for NNI #3668 @liuzhe-lz
  • - basic iframe, integrate installation script 6.29 7.14 JupyterLab extension #3954
• P0 - Add test for shared storage (PR Add shared storage integration test #3455) @acured / @QuanluZhang (NFS storage)
  • pipeline remote linux-linux timeout @ultmaster @SparkSnail
  • add NFS test 7.6 7.7 add nfs shared storage integration test #3913
• P0 - (discussion needed) design an API to share data loading/preprocessing among trials (Did every trail need to load data? #3294 )
  • design discussion
  • design review @acured 7.2
  • follow-up discussion 7.9 7.12
  • P2 - implementation
• P0 - View experiment from experiment folder @SparkSnail 6.25 Support view/resume experiment from external folder #3870
• P0 - Make reuse default @SparkSnail
  • P0 - Hybrid kubeflow support (migrate kubeflow to environment service) @SparkSnail 7.2 7.5 Support kubeflow reuse mode #3919
  • P1 - support storage service for different training services, review and implementation
  • P1 - add more test for remote mode training service
  • P2 - Frameworkcontroller support @SparkSnail
• P0 - update PT support @SparkSnail 7.9 7.12
• P0 - update quickstart doc to config v2 @kvartet Update Quickstart and remove autoCompletion in the doc #3861
• P0 - update doc for launch from python: API reference @kvartet 6.29 Update Python API doc #3885 @ultmaster
• P0 - support view trial log on webui for experiment view @SparkSnail Fix trainingService initialize error in View mode #3872
• P0 - fix path escape error in windows (issue nni's mnist-pytorch example runs failed #3869) @liuzhe-lz @acured @J-shang Escape environment variable PATH for Windows scripts #3898
  • P(-1) - fix nnimanager ut framework @liuzhe-lz 7.12 "Fix" NNI manager UT #3933
  • P(-1) - fix nnimanager ut test cases: ?, ?, ? @liuzhe-lz "Fix" NNI manager UT #3933
  • add back removed ut
• P1 - (design) backend design for project/management @acured 7.5
• P0 - update doc for exp config v2 @liuzhe-lz 7.12 7.14 7.19 Update docs to v2 config #3966
• P0 - fix training service reuse mode bug @hzhua Fix bug: GPUs are not released in Reusable Training Service #3941
• P0 - fix IP address obtaining bug: EBADF, in reuse mode? @liuzhe-lz 7.19

WebUI

• P0 - node upgrade @liuzhe-lz (Bump node.js version to v16 #3828 pr ready)
  • Change IP detection to modern way (Change NNI manager IP detection method and remove netifaces dependency #3860)
• P0 - demo: Experiment/Project Management @Lijiaoa 6.25
• P0 - Show dict intermediate result in the dialog of single trial intermediate result @Lijiaoa 7.2 [webui] Support dict intermediate metrics(only for number type) #3895
• P0 - Adaptive UI (height) for high resolution, current window for high resolution need improvement @JunzhuZhang @yangou1988 @Lijiaoa
  • fix bug in table search 7.14 fix search broken bug with incomplete filter  #3935
  • overview page: blank at the bottom
  • align issue in overview page
  • support searching customized string in the search box

Hyper-parameter tuning

• P0 - Test FLAML with HPO benchmark @xiaowu0162 6.25
• P1 - Improve HPO benchmarks, add more concrete conclusions @xiaowu0162 7.12 HPO Benchmark Fixes and New Features #3925
  • add one benchmark and one search space 7.14 HPO Benchmark Fixes and New Features #3925
• P1 - support nested search space @kvartet 6.28 discussion meeting
• P1 - support time limit early stop algorithm (Early trial stop based on some time limit #3740) @acured 6.25 Add maxTrialDuration #3863
• P1 - Improve hyperband, PBT with shared storage (dependency on shared storage implementation) @acured
• P1 - upgrade cuda, config_space, docker image @liuzhe-lz 7.21 Bump Python dependencies #3976
• P1 - search space type validation, check hyper-parameter's type, if not support expose warning @liuzhe-lz 7.21 Add search space validation for choice types #3975
• P1 - grid search does not support _name for nested search space (update doc) @kvartet
• P1 - evolution tuner does not mutate sub search space for nested search space @kvartet

Pipeline

• P0 - support python 3.9 in pipeline (wait for pytorch upgrade from 1.6 to 1.9) @ultmaster Use python 3.9 on pipeline #3881
  • fix local pipeline nnimanager ut (nnimanager test can be added back, ipc test cannot be added back) @J-shang Add nnimanager ut back #3970
  • P2 - write ipc test
• P0 - code coverage @ultmaster 6.25
• P0 - fix pipeline error: disk full, docker images are in primary disk
  • update crontab command @J-shang
  • config docker image place @SparkSnail
• P0 - fix windows pipeline @ultmaster Fix pipeline dependency #3926
  • fix linux
• P0 - fix and improve pipeline
  • fast test: lint, UT, simple IT, timeout: (1) blocked by certain test (2) real timeout -> upgrade machine @ultmaster Extend windows test timeout #3962
  • nnimanager ut, nni-ci-gpu-03 failed, maybe due to this machine's environment, cannot be executed through daemon, need reimage -> create a new vm
  • review speedup test, why it requires large memory @J-shang

Stretch goals

NAS

• P1 - Evaluate one-shot strategies on NAS benchmarks @ultmaster
• P1 - Further improve the logic of graph generation and code generation (Further integrated with TorchScript) @QuanluZhang
• P1 - Test classic nas tuners on NNI NAS benchmark and report evaluation results @ultmaster
• P2 - Migrate Cream from NAS v1.0 to Retiarii framework @ yuhangchen
• P2 - refactor of Retiarii execution engine @ultmaster
  • support local debug mode for pure-python execution engine
  • support weight transfer from supernet to submodel
  • support export of top model for pure-python execution engine
  • P2 - refactor of "self._cur_samples" in mutators
  • P2 - support validation for other strategies
• P1 - Review strategy/experiment/rest stop condition @QuanluZhang
• P1 - support graph-based NAS algorithms @ultmaster @skeletondyh
• P1 - Deal with strategy failure

model compression

• P1 - MixedMaskerPruner @J-shang
• P2 - Refactor of model graph generation (as an independent component) @zheng-ningxin

nnimanager

• P2 - Mask confidential config fields (depends on prev items) @liuzhe-lz
• P2 - nnictl experiment export improvements @J-shang
• P2 - internal tuner APIs for improving tuner debuggability, allow debugger tool @liuzhe-lz
• P2 - nnictl auto-completion @liuzhe-lz
• P2 - Dependency-aware supported in iterative pruner (discussion needed)
• P2 - Add full test (l2filter, Taylor, NetAdapt, SimulatedAnnealing, AutoCompress, AMC, Sensitivity, ADMM, Lottery Ticket)
• P1 - reuse mode: ADL kubeflow support @SparkSnail
• P2 - add/del tag of experiments (pending support from rest server)
• UI: support experiment resume/view/stop on webUI
• P0 - (discussion needed) how to support specifying initial configs (Can we set default values of hyper parameters for HPO? #3801 ) @ultmaster
• P2 - (discussion needed) support constraints on hyper-parameters @ultmaster
• P2 - handle all reject errors @J-shang

[QuanluZhang]

From Ali PAI team

• global sort, importance sort on all layers
• QAT improvement, PTQ initialization (set scale, zero point), post-training quantization
• new model compression algorithms (e.g., Bert), model-specific compression (not that generic), sparsifying training and sparsifying operator

[QuanluZhang]

bugbash

Retiarii & NAS

• P0 - NAS visualization @ultmaster 6.29 ([Retiarii] Visualization #3878) tester: @jonsnows
• P0 - Add examples for NAS-Bench-series @ultmaster tester: @skeletondyh
  • 101 6.25 [Retiarii] NAS-Bench-101 #3871
  • 201 [Retiarii] NAS-Bench-201 #3920
• P0 - Support one system-level optimization: CSE optimization to deduplicate computation @hzhua pr [Retiarii] cross-graph optimization: device placement and input deduplication #3202 tester: @QuanluZhang
• P0 - Improve training service to support CSE optimization @hzhua extend reusable training service to support placement constraint #3897 tester: @QuanluZhang
• P0 - Searching for activation functions. @QuanluZhang 6.24 6.28 ([Retiarii] support hypermodule: autoactivation #3868 ) tester: @ultmaster
• P1 - Support PyTorch 1.8/1.9 @QuanluZhang 7.14 [retiarii] support torch 1.8 and 1.9 #3937 tester: @ultmaster
• P0 - Deal with retiarii experiment exit failure @QuanluZhang 7.1 7.2 fix multi-thread event handling issue #3899 tester: @ultmaster
• P0 - support nn-meter in multi-trial nas @ultmaster @jianyu [DO NOT SQUASH] Support nn-Meter in Retiarii framework #3938 tester: @zheng-ningxin

Model Compression

• P0 - Model Speedup Refactor @zheng-ningxin pr Model Speedup Refactor #3462 tester: @J-shang
• P1 - Add more examples for model speedup @zheng-ningxin Add several speedup examples #3880 tester: @linbinskn
• P0 - Add test for TensorRT @linbinskn 6.25 @J-shang tester: @xiaowu0162
• P0 - Global sort for taylor @linbinskn 7.2 [Model Compression] Add global sort for taylor pruner #3896 tester: @xiaowu0162
• P0 - fix exclude not support in partial compressor configlist @J-shang pr ready add exclude config validate in compressor #3815 tester: @xiaowu0162
• P1 - Add one compression algorithm for Bert/Transformer @xiaowu0162 tester: @J-shang
• observer quantizer Add post training observer_quantizer #3915 tester: @linbinskn
• batchnorm folding in quantizers, Add batch normalization folding to QAT quantizer #3911 tester: @linbinskn
• test model compression quick start: activation, taylor, agp, admm tester: @skeletondyh

Training service & NNI manager && nnictl

• P0 - file name validation, from error to warning and improve warning message @liuzhe-lz (Change file name validation to warning #3843) tester: @acured
• P0 - JuypterLab support - JupyterLab support for NNI #3668 @liuzhe-lz tester: @SparkSnail
  • - basic iframe, integrate installation script 6.29 7.14 JupyterLab extension #3954
• P0 - test shared storage (PR Add shared storage integration test #3455) @acured / @QuanluZhang (NFS storage)
  • local, remote, tester: @QuanluZhang
  • aml tester: @SparkSnail
  • pai tester: @QuanluZhang
• P0 - View experiment from experiment folder @SparkSnail 6.25 Support view/resume experiment from external folder #3870 tester: @liuzhe-lz
• P0 - Make reuse default @SparkSnail
  • P0 - Hybrid kubeflow support (migrate kubeflow to environment service) @SparkSnail 7.2 7.5 Support kubeflow reuse mode #3919 tester: @acured
• P0 - update PT support @SparkSnail 7.9 7.12 tester: @SparkSnail
• P0 - update quickstart doc to config v2 @kvartet Update Quickstart and remove autoCompletion in the doc #3861 tester: @QuanluZhang
• P0 - update doc for launch from python: API reference @kvartet 6.29 Update Python API doc #3885 @ultmaster tester: @QuanluZhang
• P0 - support view trial log on webui for experiment view @SparkSnail Fix trainingService initialize error in View mode #3872 tester: @liuzhe-lz , test2: @ultmaster
• P0 - fix path escape error in windows (issue nni's mnist-pytorch example runs failed #3869) @liuzhe-lz @acured @J-shang Escape environment variable PATH for Windows scripts #3898 tester: @ultmaster
• P0 - fix training service reuse mode bug @hzhua Fix bug: GPUs are not released in Reusable Training Service #3941 tester: @acured
• P0 - fix IP address obtaining bug: EBADF, in reuse mode? @liuzhe-lz 7.19 tester: @acured
  • Change IP detection to modern way (Change NNI manager IP detection method and remove netifaces dependency #3860)

WebUI

• P0 - Show dict intermediate result in the dialog of single trial intermediate result @Lijiaoa 7.2 [webui] Support dict intermediate metrics(only for number type) #3895 tester: @liuzhe-lz
• fix bug in table search 7.14 fix search broken bug with incomplete filter  #3935 tester: @liuzhe-lz
• test tensorboard error message tester: @liuzhe-lz

Hyper-parameter tuning

• P1 - Improve HPO benchmarks, add more concrete conclusions @xiaowu0162 7.12 HPO Benchmark Fixes and New Features #3925
  • add one benchmark and one search space 7.14 HPO Benchmark Fixes and New Features #3925 tester: @jonsnows
• P1 - support time limit early stop algorithm (Early trial stop based on some time limit #3740) @acured 6.25 Add maxTrialDuration #3863 tester: @skeletondyh
• P1 - search space type validation, check hyper-parameter's type, if not support expose warning @liuzhe-lz 7.21 Add search space validation for choice types #3975 tester: @xiaowu0162

[QuanluZhang]

fix pipeline:

• integration test windows to linux @ultmaster
• integration test pai linux @liuzhe-lz @SparkSnail
• gpu-04 reconfig disks @liuzhe-lz @ultmaster
• integration test hybrid @SparkSnail
• full test linux @ultmaster
• full test windows @QuanluZhang