compression benchmark #2742

suiguoxin · 2020-07-27T06:33:48Z

Filter pruning experiment with : SimulatedAnnealing, NetAdapt, AutoCompress, L1Filter, L2Filter, FPGMPruner on cifar10 with resnet18, resnet50 and vgg16.

This #PR includes the following contents:

experiment result presentation & analysis
source code and instruction fore re-implementation
experiment result in json format

For ActivationAPoZRankFilterPruner, ActivationMeanRankFilterPruner & AGPPruner, I plan to add them in this benchmark after refactoring.

colorjam · 2020-07-28T12:46:01Z

are .json files necessary? suggest merge them into one file

chicm-ms · 2020-07-30T05:44:53Z

docs/en_US/Compressor/Benchmark.md

@@ -0,0 +1,88 @@
+To provide an initial insight into the performance of various channel pruning algorithms, 


Suggest to use term channel pruning and filter pruning to represent different pruning methods:
channel pruning: prune input channel
filter pruning: prune output channel
Our current pruner implemented as filter pruning, we may support channel pruning in the future.

Suggest to use term channel pruning and filter pruning to represent different pruning methods:
channel pruning: prune input channel
filter pruning: prune output channel
Our current pruner implemented as filter pruning, we may support channel pruning in the future.

Thx, fixed.

chicm-ms · 2020-07-30T06:08:34Z

examples/model_compress/auto_pruners_torch.py

-        print('Speed up model saved to %s', args.experiment_data_dir)
-
-    with open(os.path.join(args.experiment_data_dir, 'performance.json'), 'w+') as f:
+    with open(os.path.join(args.experiment_data_dir, 'result.json'), 'w+') as f:


Can we save all results into one json? maybe use a key to identify different experiment?

Can we save all results into one json? maybe use a key to identify different experiment?

Merged to 3 files. One file by every dataset/model combination.

examples/model_compress/models/cifar10/resnet.py

chicm-ms · 2020-07-30T06:14:00Z

docs/en_US/Compressor/Benchmark.md

+
+CIFAR-10, ResNet50:
+
+![](../../../examples/model_compress/experiment_result/img/performance_comparison_resnet50.png)


Is it easier to read if the X axis is changed to sparsity/flops ratio ?

I don't see big difference...

chicm-ms · 2020-07-30T06:17:43Z

examples/model_compress/auto_pruners_torch.py

-            train(args, model, device, train_loader,
-                  criterion, optimizer, epoch)
-            scheduler.step()
+        if args.load_pretrained_model:


Do we have the LeNet benchmark result?

chicm-ms · 2020-07-30T06:21:07Z

examples/model_compress/auto_pruners_torch.py

+def get_input_size(dataset):
+    if dataset == 'mnist':
+        input_size = (1, 1, 28, 28)
+    elif dataset in ['cifar10', 'imagenet']:


why input size for imagenet is 32x32? do we resize the image for imagenet?

There is an error here, changed to 256*256. Imagenet experiment is not performed in this PR.

QuanluZhang · 2020-07-31T00:49:29Z

docs/en_US/Compressor/Benchmark.md

@@ -0,0 +1,88 @@
+To provide an initial insight into the performance of various channel pruning algorithms, 


better to add title for this doc, for example "Comparison of Pruning Algorithms"

better to add title for this doc, for example "Comparison of Pruning Algorithms"
Title 'Comparison of Filter Pruning Algorithms' added

QuanluZhang · 2020-07-31T00:57:02Z

docs/en_US/Compressor/Benchmark.md

+        - One-shot pruners: L1Filter, L2Filter, FPGMPruner
+    - Only **channel pruning** performances are compared here. 
+
+    For the auto-pruners, `L1FilterPruner` is used as the base algorithm. That is to say, after the sparsities distribution among the layers is decided by the scheduling algorithm, `L1FilterPruner` is used to performn real pruning.


it is not clear what are the auto-pruners? Pruners with scheduling?

it is not clear what are the auto-pruners? Pruners with scheduling?

Yes, fixed

QuanluZhang · 2020-07-31T00:58:27Z

docs/en_US/Compressor/Benchmark.md

+* Pruners: 
+    - These pruners are included:
+        - Pruners with scheduling : SimulatedAnnealing, NetAdapt, AutoCompress
+        - One-shot pruners: L1Filter, L2Filter, FPGMPruner


better to mention how to set each layer's sparsity for these one-shot pruners

better to mention how to set each layer's sparsity for these one-shot pruners

Added here.

QuanluZhang · 2020-07-31T01:01:11Z

docs/en_US/Compressor/Benchmark.md

+
+CIFAR-10, VGG16:
+
+![](../../../examples/model_compress/experiment_result/img/performance_comparison_vgg16.png)


better to use different node types (not just different colors) for each pruner.

better to use different node types (not just different colors) for each pruner.

Fixed

QuanluZhang · 2020-07-31T01:03:31Z

docs/en_US/Compressor/Benchmark.md

+From the experiment result, we get the following conclusions:
+
+* Given the constraint on the number of parameters, the pruners with scheduling ( `AutoCompress` , `SimualatedAnnealing` ) performs better than the others when the constraint is strict. However, they have no such advantage in FLOPs/Performances comparison since only number of parameters constraint is considered in the optimization process; 
+* The basic algorithms `L1FilterPruner` , `L2FilterPruner` , `FPGMPruner` performs very similarly in these experiments; 


good summary. for this one, can I say for some model/dataset one-shot pruners perform similar to the pruners with scheduling, even though they set the same sparsity for every layer?

good summary. for this one, can I say for some model/dataset one-shot pruners perform similar to the pruners with scheduling, even though they set the same sparsity for every layer?

We cannot say pruners with scheduling and basic pruners are similar generally because they are different given different evaluation metrics.

Isn't FLOPs related to just filter number and channel number(which are actually the same thing)? And filter number also represents parameters. How does it not improve FLOPs? I can imagine some architecture like xception won't improve because filter number reduction may not affect channel number. But it doesn't make sense for VGG16. Is it the same situation? Please correct me if I am wrong 😄

Isn't FLOPs related to just filter number and channel number(which are actually the same thing)? And filter number also represents parameters. How does it not improve FLOPs? I can imagine some architecture like xception won't improve because filter number reduction may not affect channel number. But it doesn't make sense for VGG16. Is it the same situation? Please correct me if I am wrong 😄

Thanks for your reply. FLOPs is also related to image resolution, the input size of different layers are different.

Oh thanks for reminding me. Aren't Input size/feature map size almost the same size no matter which pruner? If so, the FLOPs should be correspond to params.

QuanluZhang · 2020-07-31T01:04:29Z

docs/en_US/Compressor/Benchmark.md

+
+### Implementation Details
+
+* The experiment results are all collected with the default configuration of the pruners in nni.


what do you mean by 'default configuration'?

what do you mean by 'default configuration'?

Explained.

QuanluZhang · 2020-07-31T01:26:55Z

docs/en_US/model_compression.rst

@@ -22,5 +22,6 @@ For details, please refer to the following tutorials:
    Automatic Model Compression <Compressor/AutoCompression>
    Model Speedup <Compressor/ModelSpeedup>
    Compression Utilities <Compressor/CompressionUtils>
+    Compression Benchmark <Compressor/Benchmark>


this is not benchmark, it is benchmark result. So better to move it to Use Cases and Solutions under Performance measurement, comparison and analysis

this is not benchmark, it is benchmark result. So better to move it to Use Cases and Solutions under Performance measurement, comparison and analysis

Exactly, thanks

QuanluZhang · 2020-07-31T01:30:42Z

examples/model_compress/experiment_result/analyze.py

@@ -0,0 +1,93 @@
+import argparse


the folder name experiment_result can be changed to comparison_of_pruners

the folder name experiment_result can be changed to comparison_of_pruners

ok, thx

suiguoxin · 2020-08-03T12:19:12Z

are .json files necessary? suggest merge them into one file

Merged to 3 files. One file by every dataset/model combination.

(cherry picked from commit accb40f)

lianxintao · 2020-09-29T09:42:55Z

Hello, To use auto_pruners_torch.py in NNI, many settings need to pass argparse.ArgumentParser, can you provide these setting lists for pruners such as AutoCompressPruner, L1FilterPruner, etc., so that the experimental results can be reproduced on NNI. Thank you.

suiguoxin · 2020-09-30T01:40:42Z

Hello, To use auto_pruners_torch.py in NNI, many settings need to pass argparse.ArgumentParser, can you provide these setting lists for pruners such as AutoCompressPruner, L1FilterPruner, etc., so that the experimental results can be reproduced on NNI. Thank you.

Thanks for your question. To reproduce the results that we present in nni, just use the default config for each pruner. Please refer to Implementation Details. Notice that the performances of your original models (model without pruning) may slightly vary from ours.

lianxintao · 2020-10-10T01:22:01Z

Thank you for your reply, but the results of this experiment are disappointing. I wonder if you have conducted similar experiments on more complex tasks such as target detection, semantic segmentation.Is it possible that the experimental results will be different for more complex tasks.

QuanluZhang · 2020-10-10T01:27:28Z

@lianxintao , we agree that the benchmarking results can be further enriched and improved. We will keep improving it. We highly encourage external contributors to contribute more benchmarking results and good-performing compression algorithms.

lianxintao · 2020-10-12T03:45:44Z

@lianxintao , we agree that the benchmarking results can be further enriched and improved. We will keep improving it. We highly encourage external contributors to contribute more benchmarking results and good-performing compression algorithms.
Oh, I see, thanks for your reply.

suiguoxin added 5 commits July 27, 2020 06:26

add compression benchmark

4d02859

fix typo

0564d04

explain AutoCompress result reference

d0003d2

add benchmark in doctree

2413636

fix typo

59b60ef

scarlett2018 mentioned this pull request Jul 29, 2020

July iteration plan #2608

Closed

66 tasks

ultmaster added the model compression label Jul 29, 2020

chicm-ms reviewed Jul 30, 2020

View reviewed changes

examples/model_compress/models/cifar10/resnet.py Show resolved Hide resolved

chicm-ms reviewed Jul 30, 2020

View reviewed changes

QuanluZhang requested review from colorjam and QuanluZhang July 30, 2020 15:34

QuanluZhang reviewed Jul 31, 2020

View reviewed changes

suiguoxin added 2 commits August 3, 2020 12:11

merge experiment result to 3 json files

4ea759e

Merge branch 'benchmark' of github.com:suiguoxin/nni into benchmark

c3419df

suiguoxin added 2 commits August 3, 2020 12:25

update folder name

eb6f8ce

refine doc

ee2a6f3

chicm-ms approved these changes Aug 4, 2020

View reviewed changes

colorjam approved these changes Aug 4, 2020

View reviewed changes

suiguoxin requested a review from QuanluZhang August 6, 2020 07:06

QuanluZhang approved these changes Aug 10, 2020

View reviewed changes

ultmaster merged commit accb40f into microsoft:master Aug 11, 2020

LovPe pushed a commit to LovPe/nni that referenced this pull request Aug 17, 2020

compression benchmark (microsoft#2742)

6a26e44

(cherry picked from commit accb40f)

suiguoxin deleted the benchmark branch August 20, 2020 08:45

		@@ -0,0 +1,88 @@
		To provide an initial insight into the performance of various channel pruning algorithms,


		CIFAR-10, ResNet50:

		![](../../../examples/model_compress/experiment_result/img/performance_comparison_resnet50.png)


		CIFAR-10, VGG16:

		![](../../../examples/model_compress/experiment_result/img/performance_comparison_vgg16.png)


		### Implementation Details

		* The experiment results are all collected with the default configuration of the pruners in nni.

compression benchmark #2742

compression benchmark #2742

Conversation

suiguoxin commented Jul 27, 2020 • edited Loading

colorjam commented Jul 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suiguoxin commented Aug 3, 2020

lianxintao commented Sep 29, 2020 • edited Loading

suiguoxin commented Sep 30, 2020

lianxintao commented Oct 10, 2020

QuanluZhang commented Oct 10, 2020

lianxintao commented Oct 12, 2020

suiguoxin commented Jul 27, 2020 •

edited

Loading

colorjam commented Jul 28, 2020 •

edited

Loading

lianxintao commented Sep 29, 2020 •

edited

Loading