[ACCV2022] Network Pruning via Feature Shift Minimization

This is the official code for our ACCV22 paper: Network Pruning via Feature Shift Minimization.

Authors: Yuanzhi Duan, Yue Zhou, Peng He, Qiang Liu, Shukai Duan, Xiaofang Hu.

The diagram of feature shift is:

• 

Environments

The code has been tested in the following environments:

• Python 3.8
• PyTorch 1.8.1
• cuda 10.2
• torchsummary, torchvision, thop, scipy, sympy

Pre-trained Models

Vgg-16 | ResNet56 | GoogLeNet | ResNet50

Running Code

The settings are listed below:

1. VGGNet

Compression Rate	Flops($\downarrow $)	Params($\downarrow $)	Accuracy
[0.3]*5+[0.35]*3+[0.8]*5	106.8M(66.0%)	2.05M(86.3%)	93.73%
[0.5]*8+[0.8]*5	59.7M(81.0%)	1.41M(90.6%)	92.86%

#VGGNet
#All run scripts can be cut-copy-paste from run.bat or run.sh.
python main.py \
--arch vgg_16_bn \
--resume [pre-trained model dir] \
--compress_rate [0.3]*5+[0.35]*3+[0.8]*5 \
--num_workers 1 \
--epochs 1 \
--lr 0.001 \
--lr_decay_step 1 \
--weight_decay 0. \
--data_dir [dataset dir] \
--dataset CIFAR10 \
--save_id 1 

python main.py \
--arch vgg_16_bn \
--from_scratch True \
--resume final_pruned_model/vgg_16_bn_1.pt \
--num_workers 1 \
--epochs 200 \
--gpu 0 \
--lr 0.01 \
--lr_decay_step 100,150 \
--weight_decay 0. \
--data_dir [dataset dir] \
--dataset CIFAR10 \
--save_id 1 

2. ResNet-56

Compression Rate	Flops($\downarrow $)	Params($\downarrow $)	Accuracy
[0.]+[0.6,0.]*9+[0.6,0.]*9+[0.4,0.]*9	61.7M(51.2%)	0.48M(43.6%)	93.58%
[0.]+[0.7,0.]*9+[0.7,0.]*9+[0.7,0.]*9	40.2M(68.2)	0.27M(68.5)	92.76%

#ResNet-56
python main.py \
--arch resnet_56 \
--resume [pre-trained model dir] \
--compress_rate [0.]+[0.6,0.]*9+[0.6,0.]*9+[0.4,0.]*9 \
--num_workers 1 \
--epochs 1 \
--lr 0.001 \
--lr_decay_step 1 \
--weight_decay 0. \
--data_dir [dataset dir] \
--dataset CIFAR10 \
--save_id 1 

python main.py \
--arch resnet_56 \
--from_scratch True \
--resume final_pruned_model/resnet_56_1.pt \
--num_workers 1 \
--epochs 300 \
--gpu 0 \
--lr 0.01 \
--lr_decay_step 150,225 \
--weight_decay 0.0005 \
--data_dir [dataset dir] \
--dataset CIFAR10 \
--save_id 1 

3. GoogLeNet

Compression Rate	Flops($\downarrow $)	Params($\downarrow $)	Accuracy
[0.2]+[0.7]*15+[0.8]*9+[0.,0.4,0.]	0.567B(63.0%)	2.75M(55.5%)	94.72%
[0.2]+[0.9]*24+[0.,0.4,0.]	0.376B(75.4%)	2.19M(64.6%)	94.29%

#GoogLeNet
python main.py \
--arch googlenet \
--resume [pre-trained model dir] \
--compress_rate [0.2]+[0.7]*15+[0.8]*9+[0.,0.4,0.] \
--num_workers 1 \
--epochs 1 \
--lr 0.001 \
--weight_decay 0. \
--data_dir [dataset dir] \
--dataset CIFAR10 \
--save_id 1 

python main.py \
--arch googlenet \
--from_scratch True \
--resume final_pruned_model/googlenet_1.pt \
--num_workers 1 \
--epochs 200 \
--lr 0.01 \
--lr_decay_step 100,150 \
--weight_decay 0. \
--data_dir [dataset dir] \
--dataset CIFAR10 \
--save_id 1 

4. ResNet-50

Compression Rate	Flops($\downarrow $)	Params($\downarrow $)	Top-1 Acc	Top-5 Acc
[0.]+[0.2,0.2,0.2]*1+[0.65,0.65,0.2]*2+[0.2,0.2,0.2]*1+[0.65,0.65,0.2]*3+[0.2,0.2,0.2]*1+[0.65,0.65,0.2]*5+[0.2,0.2,0.1]+[0.2,0.2,0.1]*2	1.76B(57.2%)	14.6M(42.8%)	75.43%	92.45%

#ResNet-50
python main.py \
--arch resnet_50 \
--resume [pre-trained model dir] \
--data_dir [dataset dir] \
--dataset ImageNet \
--compress_rate [0.]+[0.2,0.2,0.2]*1+[0.65,0.65,0.2]*2+[0.2,0.2,0.2]*1+[0.65,0.65,0.2]*3+[0.2,0.2,0.2]*1+[0.65,0.65,0.2]*5+[0.2,0.2,0.1]+[0.2,0.2,0.1]*2 \
--num_workers 4 \
--batch_size 64 \
--epochs 1 \
--lr_decay_step 1 \
--lr 0.001 \
--weight_decay 0. \
--input_size 224 \
--save_id 1 

python main.py \
--arch resnet_50 \
--from_scratch True \
--resume finally_pruned_model/resnet_50_1.pt \
--num_workers 4 \
--epochs 100 \
--lr 0.01 \
--lr_decay_step 30,60,90 \
--batch_size 64 \
--weight_decay 0.0001 \
--input_size 224 \
--data_dir [dataset dir] \
--dataset ImageNet \
--save_id 1

Acknowledgments

Our implementation partially reuses HRank's code.