Grid Analyzer!

[tkornuta-ibm]

Finally, working mip-grid-analyzer is here!

This required to solve several issues and introduce many changes in the whole experimental pipeline, starting from:

• rethinking what we really want to show in the final csv file - fixes Rethink operation of grid-analyzer #66
• investigating the content of statistics and rethinking the desired behavior of online/offline trainers (fixes Investigate the content of statistics, rethink the desired behavior of online/offline trainers #64 )
• store termination cause (fixes Store termination cause in model checkpoint #67 ) along with testing and validation statistics (fixes Add exporting both training and validation statistics to model #74 ) in model checkpoint
• fix formatting issue in grid-analyzer(fixes Fix formatting issue in grid-analyzer #77 )
• I have also introduced fixes to all grid-workers-cpu that enables to distribute computations into several processes on CPU on MacOS (fixes grid_*_cpu not working on MAC/OSX #52 ) (WARNING! Didn't tested that on Ubuntu CPUs nor GPUs, should work, but needs testing... - creating new issues Test grid-* image classification pipeline on Ubuntu CPUs #82 and Test grid-* image classification pipeline on GPUs #83 )

In order to test the whole pipeline, I have also prepared a config for grid_trainer with LeNet5 and SimpleCNN model trained on MNIST (fixes #51 )

The use case is as follows:

• run grid-trainer on the configuration file (2 models, 1 problem, 5 repetitions)
  'mip-grid-trainer-cpu --c configs/vision/grid_trainer_mnist.yaml'

• run grid-tested on the newly created directory with containing 10 different models
  'mip-grid-tester-cpu --e --expdir ./experiments_20181109_131153'

• run grid-analyzer to aggregate all the results in a single csv file
  'mip-grid-analyzer --e --expdir ./experiments_20181109_131153'

Attaching the resulting file:

20181109_134802_grid_analysis.xlsx

(What I have described above will in fact become a part of "reproducible research: MNIST Image Classification with LeNet-5 - issue #12 )

Besides, I've made plenty of minor changes, like:

• added Ctrl+C handling to grid workers (fixes Add Ctrl+C handling to grid workers #63 )
• changed --o to --e (fixes Change --o to --e #65 )
• cleaned up some of the arguments of grid workers (partially Clean up the flags of the different workers  #25 )
• adds option to select basic trainer from both command line and config of grid_trainer_* (fixes Add basic trainer selection to config of grid_trainer_* #62)

[tkornuta-ibm]

This pull request introduces 3 alerts when merging 1ae12d3 into d8c20f1 - view on LGTM.com

new alerts:

• 3 for Unused import

---

Comment posted by LGTM.com

[vmarois]

It looks good! I haven't add the time to test it so will do so on Monday.

This PR adds a dependency over psutil.. Is there no way around it?

[tkornuta-ibm]

psutil

Sadly not, please review the solutions that I have analyzed when fixing issue #52

[vmarois]

It looks good, thanks!
I have done some doc polishing and set a few basic methods of GridAnalyzer as static ones.
I have tested the pipeline on Ubuntu CPUs (fixes #82 ) and GPUs (fixes #83 ), and it worked flawlessly.

Nonetheless, I have some remarks about the resulting csv file:

• The train_status value does not always match the actual one reported in the trainer.log. Critically, some runs are indicated as "Not converged" when they actually did. I am thinking that this happens as we do not always save the model. Thus, it is possible that we save the model mid-training, and the training status is then different than the final one

• I find the key name train_start to be somewhat confusing when compared to model_timestamp (I thought at the beginning that we were not reporting the correct timestamp). Could we clarify this?

• We should report the aggregated statistics for the test, in place of the data point for the last episode. I think this is more valuable to the user, and shouldn't be hard to implement (since we have only 1 data point for the test, and the corresponding csv file exists in all cases)

Otherwise, this is an important PR, and this is great work! 👍