Skip to content
/ awesome-dynamic-conditional-networks-cv Public

Overview of conditional computution and dynamic CNNs for computer vision, with a focus on reducing computational complexity

41 stars 4 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

thomasverelst/awesome-dynamic-conditional-networks-cv

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
README.md
README.md
 
 

Repository files navigation

Awesome Dynamic Networks and Conditional Computation



Upcoming ICML 2022 on Dynamic Neural Networks! https://dynn-icml2022.github.io/ on Friday, July 22.



Overview of conditional computation and dynamic CNNs for computer vision, focusing on reducing computational cost of existing network architectures. In contrast to static networks, dynamic networks disable parts of the network based on the input image, at inference time. This can save computations and speed up inference, for example by processing easy images with fewer operations. Note that this list mainly focuses on methods reducing the computational cost of existing models (e.g. ResNet models), and does not list all methods that use dynamic computation for custom architectures.

This list is growing every day. If a method is missing or listed incorrectly, let me know by making a GitHub issue or pull request!

Here is a list with more static and dynamic methods for efficient CNNs.

Background

Methods have three important distinguishing factors:

  • The method's architecture, e.g. skipping layers or pixels, and whether these run-or-skip decisions are the result of a separate policy network, a submodule in the network or another mechanism.
  • The way of training the policy, e.g. using reinforcement learning, the gradient estimator such as Gumbel-Softmax or a custom approach.
  • The implementation of the method, and whether the method can be executed efficiently on existing platforms (i.e. whether the method speeds up inference, or only reduces the theoretical amount of computations)

Metrics: Most methods demonstrate performance with the reduction in computations (i.e. measured in floating point operations, FLOPS) compared to the loss in accuracy. Methods typically show figures where baseline models of different complexities (e.g. by reducing the number of channels) are compared to the method applied to the largest model with different cost savings.

Note that many works express computational complexity in FLOPS, even though the given numbers are actually multiply-accumulate operations (MACs), and GMACs = 0.5 * GFLOPs (see sovrasov/flops-counter.pytorch#16 ). Some recent works therefore use GMAC instead of GFLOP to avoid ambiguity.

Tags used below: Note: tags are incomplete

  • VID: Video processing

Surveys / overviews

  • Dynamic Neural Networks: A Survey (Arxiv 2021) [pdf] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, Yulin Wang

Methods

Depth-based methods

Early-exit methods have separate output branches to apply more or fewer layers.

  • BranchyNet: Fast inference via early exiting from deep neural networks (ICPR2016) [pdf] [chainer]
    Teerapittayanon S, McDanel B, Kung HT
  • Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognition (DATE2016) [pdf]
    P. Panda, A. Sengupta, and K. Roy
  • Adaptive Neural Networks for Efficient Inference (ICML2017) [pdf] [GitHub no code]
    T. Bolukbasi, J. Wang, O. Dekel, and V. Saligrama
  • Dynamic computational time for visual attention (ICCV2017 workshop) [pdf] [torch lua]
    Li, Z., Yang, Y., Liu, X., Zhou, F., Wen, S. and Xu, W.
  • DynExit: A Dynamic Early-Exit Strategy for Deep Residual Networks (SiPS2019) [pdf]
    M. Wang, J. Mo, J. Lin, Z. Wang, and L. Du
  • Improved Techniques for Training Adaptive Deep Networks (ICCV2019) [pdf] [Pytorch]
    H. Li, H. Zhang, X. Qi, Y. Ruigang, and G. Huang
  • Early-exit convolutional neural networks (thesis 2019) [pdf]
    E. Demir
  • Efficient adaptive inference for deep convolutional neural networks using hierarchical early exits (Pattern Recognition 2020) [pdf]
    N. Passalis, J. Raitoharju, A. Tefas, and M. Gabbouj
  • Triple wins: Boosting accuracy, robustness and efficiency together by enabling input-adaptive inference (ICLR2020) [pdf] [pytorch]
    Hu TK, Chen T, Wang H, Wang Z.
  • FrameExit: Conditional Early Exiting for Efficient Video Recognition [pdf] Ghodrati, A., Bejnordi, B. E., & Habibian, A.
    [VID]

Skipping layers conditioned on the input image. For instance, easy images require fewer layers than complex ones:

  • Adaptive Computation Time for Recurrent Neural Networks (NIPS 2016 Deep Learning Symposium) [pdf] [unofficial pytorch]
    A. Graves
  • Convolutional Networks with Adaptive Inference Graphs (ECCV2018) [pdf] [Pytorch]
    A. Veit and S. Belongie
  • SkipNet: Learning Dynamic Routing in Convolutional Networks (ECCV2018) [pdf] [Pytorch]
    X. Wang, F. Yu, Z.-Y. Dou, T. Darrell, and J. E. Gonzalez
  • BlockDrop: Dynamic Inference Paths in Residual Networks (CVPR2018) [pdf] [Pytorch]
    Zuxuan Wu*, Tushar Nagarajan*, Abhishek Kumar, Steven Rennie, Larry S. Davis, Kristen Grauman, and Rogerio Feris
  • Dynamic Multi-path Neural Network (Arxiv2019) [pdf]
    Su, Y., Zhou, S., Wu, Y., Su, T., Liang, D., Liu, J., Zheng, D., Wang, Y., Yan, J. and Hu, X.
  • Energynet: Energy-efficient dynamic inference (2018) [pdf]
    Wang, Yue, et al.
  • Dual dynamic inference: Enabling more efficient, adaptive and controllable deep inference (IEEE Journal of Selected Topics in Signal Processing 2020) [pdf]
    Wang Y, Shen J, Hu TK, Xu P, Nguyen T, Baraniuk RG, Wang Z, Lin Y.
  • CoDiNet: Path Distribution Modeling with Consistency and Diversity for Dynamic Routing (TPAMI 2021) [pdf]

Executes some layers multiple times ('recursively') based on complexity:

  • IamNN: Iterative and Adaptive Mobile Neural Network for Efficient Image Classification (ICLR2018 Workshop) [pdf]
    S. Leroux, P. Molchanov, P. Simoens, B. Dhoedt, T. Breuel, and J. Kautz
  • Dynamic recursive neural network (CPVR2019) [pdf]
    Guo, Q., Yu, Z., Wu, Y., Liang, D., Qin, H., and Yan, J.

Channel-based methods

Channel-based methods execute specific channels to reduce computational complexity.

  • Estimating or propagating gradients through stochastic neurons for conditional computation [pdf]
    Bengio Y, Léonard N, Courville A.

  • Runtime Neural Pruning (NIPS2017) [pdf]
    J. Lin, Y. Rao, J. Lu, and J. Zhou

  • Dynamic Channel Pruning: Feature Boosting and Suppression (Arxiv2018) [pdf] [tensorflow] [unoffical pytorch]
    X. Gao, Y. Zhao, Ł. Dudziak, R. Mullins, and C. Xu.

  • Channel Gating Neural Networks (NIPS2019) [pdf] [pytorch]
    W. Hua, Y. Zhou, C. M. De Sa, Z. Zhang, and G. E. Suh

  • You Look Twice: GaterNet for Dynamic Filter Selection in CNNs (CVPR2019) [pdf]
    Z. Chen, Y. Li, S. Bengio, and S. Si

  • Runtime Network Routing for Efficient Image Classification (TPAMI2019) [pdf]
    Y. Rao, J. Lu, J. Lin, and J. Zhou

  • Dynamic Neural Network Channel Execution for Efficient Training (BMVC2019) [pdf]
    S. E. Spasov and P. Lio

  • Learning Instance-wise Sparsity for Accelerating Deep Models (IJCAI2019) [pdf]
    Liu C, Wang Y, Han K, Xu C, Xu C.

  • Batch-Shaping for Learning Conditional Channel Gated Networks (ICLR2020) [pdf]
    BE Bejnordi, T Blankevoort, M Welling

  • Dynamic slimmable network (CVPR2021) [pdf] [pytorch]
    Li, Changlin, et al.

  • Dynamic Slimmable Denoising Network. (2021) [pdf] Jiang, Zutao, Changlin Li, Xiaojun Chang, Jihua Zhu, and Yi Yang

  • DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers (2021) [pdf] Li, C., Wang, G., Wang, B., Liang, X., Li, Z., & Chang, X.

  • Borrowing from yourself: Faster future video segmentation with partial channel update (2022) [pdf]

  • Multi-dimensional dynamic model compression for efficient image super-resolution (WACV2022) [pdf]

Spatial methods