counting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Counting for MicroNet Challenge Score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Altough we thoroughly specified the calculation results of our model in `readme.md` file, we double checked the score using refactored counting code which has uploaded by the organizers: [official counting code](https://github.com/google-research/google-research/blob/master/micronet_challenge/counting.py)\n",
    "\n",
    "We defined our `read model` module and `counting module` for newly defined operations. Since our backbone architecture is following EfficientNet, which was the example of official counting code, we examined our counting code for EfficientNet.\n",
    "\n",
    "The result from our counting code was same with official counting code for EfficientNet. Below are the results for the score for our model and EfficientNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. EfficientNet-b0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz to ./cifar-100-python.tar.gz\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████▉| 168910848/169001437 [03:25<00:00, 1371047.06it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "EfficientNet(\n",
       "  (_conv_stem): MaskedConv2dStaticSamePadding(\n",
       "    3, 24, kernel_size=(3, 3), stride=(2, 2), bias=False\n",
       "    (static_padding): ZeroPad2d(padding=(0, 1, 0, 1), value=0.0)\n",
       "  )\n",
       "  (_bn0): BatchNorm(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (_blocks): ModuleList(\n",
       "    (0): MBConvBlock(\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        32, 32, kernel_size=(3, 3), stride=[1, 1], groups=32, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)\n",
       "      )\n",
       "      (_bn1): BatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        32, 8, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        8, 32, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        32, 16, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (1): MBConvBlock(\n",
       "      (_expand_conv): MaskedConv2dStaticSamePadding(\n",
       "        16, 96, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn0): BatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        96, 96, kernel_size=(3, 3), stride=[2, 2], groups=96, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(0, 1, 0, 1), value=0.0)\n",
       "      )\n",
       "      (pooling): AvgPool2d(kernel_size=3, stride=(2, 2), padding=1)\n",
       "      (_bn1): BatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        96, 4, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        4, 96, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        96, 24, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (2): MBConvBlock(\n",
       "      (_expand_conv): MaskedConv2dStaticSamePadding(\n",
       "        24, 144, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn0): BatchNorm(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        144, 144, kernel_size=(3, 3), stride=(1, 1), groups=144, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)\n",
       "      )\n",
       "      (_bn1): BatchNorm(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        144, 6, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        6, 144, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        144, 24, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (3): MBConvBlock(\n",
       "      (_expand_conv): MaskedConv2dStaticSamePadding(\n",
       "        24, 144, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn0): BatchNorm(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        144, 144, kernel_size=(5, 5), stride=[2, 2], groups=144, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(1, 2, 1, 2), value=0.0)\n",
       "      )\n",
       "      (pooling): AvgPool2d(kernel_size=3, stride=(2, 2), padding=1)\n",
       "      (_bn1): BatchNorm(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        144, 6, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        6, 144, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        144, 40, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(40, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (4): MBConvBlock(\n",
       "      (_expand_conv): MaskedConv2dStaticSamePadding(\n",
       "        40, 240, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn0): BatchNorm(240, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        240, 240, kernel_size=(5, 5), stride=(1, 1), groups=240, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)\n",
       "      )\n",
       "      (_bn1): BatchNorm(240, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        240, 10, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        10, 240, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        240, 40, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(40, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (5): MBConvBlock(\n",
       "      (_expand_conv): MaskedConv2dStaticSamePadding(\n",
       "        40, 240, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn0): BatchNorm(240, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        240, 240, kernel_size=(3, 3), stride=[2, 2], groups=240, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(0, 1, 0, 1), value=0.0)\n",
       "      )\n",
       "      (pooling): AvgPool2d(kernel_size=3, stride=(2, 2), padding=1)\n",
       "      (_bn1): BatchNorm(240, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        240, 10, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        10, 240, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        240, 80, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(80, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (6): MBConvBlock(\n",
       "      (_expand_conv): MaskedConv2dStaticSamePadding(\n",
       "        80, 480, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn0): BatchNorm(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        480, 480, kernel_size=(3, 3), stride=(1, 1), groups=480, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)\n",
       "      )\n",
       "      (_bn1): BatchNorm(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        480, 20, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        20, 480, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        480, 80, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(80, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (7): MBConvBlock(\n",
       "      (_expand_conv): MaskedConv2dStaticSamePadding(\n",
       "        80, 480, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn0): BatchNorm(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        480, 480, kernel_size=(3, 3), stride=(1, 1), groups=480, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)\n",
       "      )\n",
       "      (_bn1): BatchNorm(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        480, 20, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        20, 480, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        480, 80, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(80, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (8): MBConvBlock(\n",
       "      (_expand_conv): MaskedConv2dStaticSamePadding(\n",
       "        80, 480, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn0): BatchNorm(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        480, 480, kernel_size=(5, 5), stride=[1, 1], groups=480, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)\n",
       "      )\n",
       "      (_bn1): BatchNorm(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        480, 20, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        20, 480, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        480, 112, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(112, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (9): MBConvBlock(\n",
       "      (_expand_conv): MaskedConv2dStaticSamePadding(\n",
       "        112, 672, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn0): BatchNorm(672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        672, 672, kernel_size=(5, 5), stride=(1, 1), groups=672, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)\n",
       "      )\n",
       "      (_bn1): BatchNorm(672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        672, 28, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        28, 672, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        672, 112, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(112, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (10): MBConvBlock(\n",
       "      (_expand_conv): MaskedConv2dStaticSamePadding(\n",
       "        112, 672, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn0): BatchNorm(672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        672, 672, kernel_size=(5, 5), stride=(1, 1), groups=672, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)\n",
       "      )\n",
       "      (_bn1): BatchNorm(672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        672, 28, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        28, 672, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        672, 112, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(112, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (11): MBConvBlock(\n",
       "      (_expand_conv): MaskedConv2dStaticSamePadding(\n",
       "        112, 672, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn0): BatchNorm(672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        672, 672, kernel_size=(5, 5), stride=[2, 2], groups=672, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(1, 2, 1, 2), value=0.0)\n",
       "      )\n",
       "      (pooling): AvgPool2d(kernel_size=3, stride=(2, 2), padding=1)\n",
       "      (_bn1): BatchNorm(672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        672, 28, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        28, 672, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        672, 192, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (12): MBConvBlock(\n",
       "      (_expand_conv): MaskedConv2dStaticSamePadding(\n",
       "        192, 1152, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn0): BatchNorm(1152, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        1152, 1152, kernel_size=(5, 5), stride=(1, 1), groups=1152, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)\n",
       "      )\n",
       "      (_bn1): BatchNorm(1152, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        1152, 48, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        48, 1152, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        1152, 192, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (13): MBConvBlock(\n",
       "      (_expand_conv): MaskedConv2dStaticSamePadding(\n",
       "        192, 1152, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn0): BatchNorm(1152, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        1152, 1152, kernel_size=(5, 5), stride=(1, 1), groups=1152, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)\n",
       "      )\n",
       "      (_bn1): BatchNorm(1152, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        1152, 48, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        48, 1152, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        1152, 192, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (14): MBConvBlock(\n",
       "      (_expand_conv): MaskedConv2dStaticSamePadding(\n",
       "        192, 1152, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn0): BatchNorm(1152, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        1152, 1152, kernel_size=(5, 5), stride=(1, 1), groups=1152, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)\n",
       "      )\n",
       "      (_bn1): BatchNorm(1152, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        1152, 48, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        48, 1152, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        1152, 192, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (15): MBConvBlock(\n",
       "      (_expand_conv): MaskedConv2dStaticSamePadding(\n",
       "        192, 1152, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn0): BatchNorm(1152, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_conv): MaskedConv2dStaticSamePadding(\n",
       "        1152, 1152, kernel_size=(3, 3), stride=[1, 1], groups=1152, bias=False\n",
       "        (static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)\n",
       "      )\n",
       "      (_bn1): BatchNorm(1152, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (_se_reduce): MaskedConv2dStaticSamePadding(\n",
       "        1152, 48, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_se_expand): MaskedConv2dStaticSamePadding(\n",
       "        48, 1152, kernel_size=(1, 1), stride=(1, 1)\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_project_conv): MaskedConv2dStaticSamePadding(\n",
       "        1152, 320, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "        (static_padding): Identity()\n",
       "      )\n",
       "      (_bn2): BatchNorm(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "  )\n",
       "  (_conv_head): MaskedConv2dStaticSamePadding(\n",
       "    320, 150, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "    (static_padding): Identity()\n",
       "  )\n",
       "  (_bn1): BatchNorm(150, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (_fc): MaskedLinear(in_features=150, out_features=1000, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import sys\n",
    "import math\n",
    "import copy\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.optim.lr_scheduler as lr_scheduler\n",
    "import torchvision.models as models\n",
    "\n",
    "from data_utils import *\n",
    "from train_tools import *\n",
    "from models import *\n",
    "from counting import *\n",
    "\n",
    "dataloaders, dataset_sizes = cifar_100_setter(batch_size=128, \n",
    "                                              valid_size=5000,\n",
    "                                              root='./',\n",
    "                                              fixed_valid=True,\n",
    "                                              autoaugment=False)\n",
    "\n",
    "blocks_args = [\n",
    "    'r1_k3_s1_e1_i32_o16_se0.25', \n",
    "    'r2_k3_s2_e6_i16_o24_se0.25', \n",
    "    'r2_k5_s2_e6_i24_o40_se0.25',  \n",
    "    'r3_k3_s2_e6_i40_o80_se0.25',  \n",
    "    'r3_k5_s1_e6_i80_o112_se0.25',  \n",
    "    'r4_k5_s2_e6_i112_o192_se0.25',  \n",
    "    'r1_k3_s1_e6_i192_o320_se0.25'\n",
    "]\n",
    "\n",
    "blocks_args, global_params = efficientnet(blocks_args=blocks_args,\n",
    "                                          resolution_coefficient=1,\n",
    "                                          width_coefficient=1, \n",
    "                                          depth_coefficient=1, \n",
    "                                          image_size=224, \n",
    "                                          num_classes=1000)\n",
    "\n",
    "model = EfficientNet(blocks_args, \n",
    "                     global_params)\n",
    "\n",
    "model.to('cuda:1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Score of EfficientNet-b0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "op_name                     inp_size   kernel_size   in channels  out channels  params(MBytes)   mults(M)    adds(M)     MFLOPS\n",
      "==================================================\n",
      "_conv_stem                       224             3             3            32           0.002      6.021     11.239     17.261\n",
      "block_0                          112            -1            32            -1           0.003      5.821     10.839     16.659\n",
      "block_1                          112            -1            16            -1           0.012     17.010     31.010     48.020\n",
      "block_2                           56            -1            24            -1           0.021     14.452     27.548     42.000\n",
      "block_3                           56            -1            24            -1           0.030      9.992     18.855     28.848\n",
      "block_4                           28            -1            40            -1           0.062     10.540     20.514     31.054\n",
      "block_5                           28            -1            40            -1           0.073      6.236     12.000     18.236\n",
      "block_6                           14            -1            80            -1           0.204      8.289     16.295     24.584\n",
      "block_7                           14            -1            80            -1           0.204      8.289     16.295     24.584\n",
      "block_8                           14            -1            80            -1           0.250     10.547     20.717     31.264\n",
      "block_9                           14            -1           112            -1           0.414     16.879     33.361     50.240\n",
      "block_10                          14            -1           112            -1           0.414     16.879     33.361     50.240\n",
      "block_11                          14            -1           112            -1           0.522     11.232     22.132     33.364\n",
      "block_12                           7            -1           192            -1           1.171     11.798     23.424     35.222\n",
      "block_13                           7            -1           192            -1           1.171     11.798     23.424     35.222\n",
      "block_14                           7            -1           192            -1           1.171     11.798     23.424     35.222\n",
      "block_15                           7            -1           192            -1           1.429     14.959     29.689     44.649\n",
      "_conv_head                         7             1           320          1280           0.822     10.129     20.133     30.262\n",
      "_avg_pooling                       7            -1          1280             1           0.000      0.001      0.061      0.062\n",
      "_fc                                1            -1          1280          1000           2.562      0.640      1.280      1.920\n",
      "total                                                                                   10.535    203.312    395.601    598.913\n",
      "##################################################\n",
      "flops: 598.9129M, params: 10.5351MBytes\n",
      "score: 0.0571 + 0.0722 = 0.1293\n"
     ]
    }
   ],
   "source": [
    "from counting import *\n",
    "\n",
    "# add_bits_base=32, since 32 bit adds count 1 add.\n",
    "# mul_bits_base=32, since multiplications with 32 bit input count 1 multiplication.\n",
    "conv_stem = {'kernel': 3, 'stride': 2, 'out_channel': 32}\n",
    "last_ops = {'out_channel': 1280, 'num_classes': 1000}\n",
    "activation = 'swish'\n",
    "input_size = 224\n",
    "use_bias = True\n",
    "\n",
    "counter = MicroNetCounter(conv_stem, blocks_args, global_params, last_ops, activation, input_size, use_bias, add_bits_base=32, mul_bits_base=32)\n",
    "\n",
    "# Constants\n",
    "INPUT_BITS = 16\n",
    "ACCUMULATOR_BITS = 32\n",
    "PARAMETER_BITS = INPUT_BITS\n",
    "SUMMARIZE_BLOCKS = True\n",
    "SPARSITY = 0.0\n",
    "\n",
    "params, flops, _, _ = counter.print_summary(SPARSITY, PARAMETER_BITS, ACCUMULATOR_BITS, INPUT_BITS, summarize_blocks=SUMMARIZE_BLOCKS)\n",
    "\n",
    "print('#'*50)\n",
    "print(\"flops: {:.4f}M, params: {:.4f}MBytes\".format(flops, params))\n",
    "print('score: {:.4f} + {:.4f} = {:.4f}'.format(flops/(10490), params/(36.5*4), flops/(10490) + params/(36.5*4)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Further Validation\n",
    "\n",
    "Following parts are definitions for named tuples in official counting code.\n",
    "\n",
    "We examined our counting code for EfficientNet to verify the result of `read model` in official code and our code is same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import collections\n",
    "\n",
    "Conv2D = collections.namedtuple(\n",
    "    'Conv2D', ['input_size', 'kernel_shape', 'strides', 'padding', 'use_bias',\n",
    "               'activation'])\n",
    "\n",
    "\"\"\"Operation definition for 2D depthwise convolution.\n",
    "Only difference compared to Conv2D is the kernel_shape[3] = 1.\n",
    "\"\"\"\n",
    "DepthWiseConv2D = collections.namedtuple(\n",
    "    'DepthWiseConv2D', ['input_size', 'kernel_shape', 'strides', 'padding',\n",
    "                        'use_bias', 'activation'])\n",
    "\n",
    "\"\"\"Operation definition for Global Average Pooling.\n",
    "Attributes:\n",
    "  input_size: int, Dimensions of the input image (square assumed).\n",
    "  n_channels: int, Number of output dimensions.\n",
    "\"\"\"\n",
    "GlobalAvg = collections.namedtuple('GlobalAvg', ['input_size', 'n_channels'])\n",
    "\n",
    "\"\"\"Operation definitions for elementwise multiplication and addition.\n",
    "Attributes:\n",
    "  input_size: int, Dimensions of the input image (square assumed).\n",
    "  n_channels: int, Number of output dimensions.\n",
    "\"\"\"\n",
    "Scale = collections.namedtuple('Scale', ['input_size', 'n_channels'])\n",
    "Add = collections.namedtuple('Add', ['input_size', 'n_channels'])\n",
    "\n",
    "\n",
    "\"\"\"Operation definitions for elementwise multiplication and addition.\n",
    "Attributes:\n",
    "  kernel_shape: list, of length 2. Shape of the weight matrix.\n",
    "  use_bias: bool, if true a bias term is added to the output.\n",
    "  activation: str, type of activation applied to the output.\n",
    "\"\"\"\n",
    "FullyConnected = collections.namedtuple(\n",
    "    'FullyConnected', ['kernel_shape', 'use_bias', 'activation'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Result of read_model for EfficientNet-b0 (from official counting code)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "official = [('_conv_stem', Conv2D(input_size=224, kernel_shape=[3, 3, 3, 32], strides=(2, 2), padding='same', use_bias=True, activation='swish')), ('block_0', [('_depthwise_conv', DepthWiseConv2D(input_size=112, kernel_shape=[3, 3, 32, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=112, n_channels=32)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 32, 8], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 8, 32], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=112, n_channels=32)), ('_project_conv', Conv2D(input_size=112, kernel_shape=[1, 1, 32, 16], strides=(1, 1), padding='same', use_bias=True, activation=None))]), ('block_1', [('_expand_conv', Conv2D(input_size=112, kernel_shape=[1, 1, 16, 96], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=112, kernel_shape=[3, 3, 96, 1], strides=(2, 2), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=56, n_channels=96)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 96, 4], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 4, 96], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=56, n_channels=96)), ('_project_conv', Conv2D(input_size=56, kernel_shape=[1, 1, 96, 24], strides=(1, 1), padding='same', use_bias=True, activation=None))]), ('block_2', [('_expand_conv', Conv2D(input_size=56, kernel_shape=[1, 1, 24, 144], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=56, kernel_shape=[3, 3, 144, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=56, n_channels=144)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 144, 6], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 6, 144], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=56, n_channels=144)), ('_project_conv', Conv2D(input_size=56, kernel_shape=[1, 1, 144, 24], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=56, n_channels=144))]), ('block_3', [('_expand_conv', Conv2D(input_size=56, kernel_shape=[1, 1, 24, 144], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=56, kernel_shape=[5, 5, 144, 1], strides=(2, 2), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=28, n_channels=144)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 144, 6], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 6, 144], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=28, n_channels=144)), ('_project_conv', Conv2D(input_size=28, kernel_shape=[1, 1, 144, 40], strides=(1, 1), padding='same', use_bias=True, activation=None))]), ('block_4', [('_expand_conv', Conv2D(input_size=28, kernel_shape=[1, 1, 40, 240], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=28, kernel_shape=[5, 5, 240, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=28, n_channels=240)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 240, 10], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 10, 240], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=28, n_channels=240)), ('_project_conv', Conv2D(input_size=28, kernel_shape=[1, 1, 240, 40], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=28, n_channels=240))]), ('block_5', [('_expand_conv', Conv2D(input_size=28, kernel_shape=[1, 1, 40, 240], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=28, kernel_shape=[3, 3, 240, 1], strides=(2, 2), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=14, n_channels=240)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 240, 10], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 10, 240], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=14, n_channels=240)), ('_project_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 240, 80], strides=(1, 1), padding='same', use_bias=True, activation=None))]), ('block_6', [('_expand_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 80, 480], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=14, kernel_shape=[3, 3, 480, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=14, n_channels=480)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 480, 20], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 20, 480], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=14, n_channels=480)), ('_project_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 480, 80], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=14, n_channels=480))]), ('block_7', [('_expand_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 80, 480], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=14, kernel_shape=[3, 3, 480, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=14, n_channels=480)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 480, 20], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 20, 480], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=14, n_channels=480)), ('_project_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 480, 80], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=14, n_channels=480))]), ('block_8', [('_expand_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 80, 480], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=14, kernel_shape=[5, 5, 480, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=14, n_channels=480)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 480, 20], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 20, 480], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=14, n_channels=480)), ('_project_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 480, 112], strides=(1, 1), padding='same', use_bias=True, activation=None))]), ('block_9', [('_expand_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 112, 672], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=14, kernel_shape=[5, 5, 672, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=14, n_channels=672)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 672, 28], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 28, 672], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=14, n_channels=672)), ('_project_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 672, 112], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=14, n_channels=672))]), ('block_10', [('_expand_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 112, 672], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=14, kernel_shape=[5, 5, 672, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=14, n_channels=672)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 672, 28], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 28, 672], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=14, n_channels=672)), ('_project_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 672, 112], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=14, n_channels=672))]), ('block_11', [('_expand_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 112, 672], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=14, kernel_shape=[5, 5, 672, 1], strides=(2, 2), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=7, n_channels=672)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 672, 28], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 28, 672], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=7, n_channels=672)), ('_project_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 672, 192], strides=(1, 1), padding='same', use_bias=True, activation=None))]), ('block_12', [('_expand_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 192, 1152], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=7, kernel_shape=[5, 5, 1152, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=7, n_channels=1152)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 1152, 48], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 48, 1152], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=7, n_channels=1152)), ('_project_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 1152, 192], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=7, n_channels=1152))]), ('block_13', [('_expand_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 192, 1152], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=7, kernel_shape=[5, 5, 1152, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=7, n_channels=1152)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 1152, 48], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 48, 1152], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=7, n_channels=1152)), ('_project_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 1152, 192], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=7, n_channels=1152))]), ('block_14', [('_expand_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 192, 1152], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=7, kernel_shape=[5, 5, 1152, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=7, n_channels=1152)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 1152, 48], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 48, 1152], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=7, n_channels=1152)), ('_project_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 1152, 192], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=7, n_channels=1152))]), ('block_15', [('_expand_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 192, 1152], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=7, kernel_shape=[3, 3, 1152, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=7, n_channels=1152)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 1152, 48], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 48, 1152], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=7, n_channels=1152)), ('_project_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 1152, 320], strides=(1, 1), padding='same', use_bias=True, activation=None))]), ('_conv_head', Conv2D(input_size=7, kernel_shape=[1, 1, 320, 1280], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_avg_pooling', GlobalAvg(input_size=7, n_channels=1280)), ('_fc', FullyConnected(kernel_shape=[1280, 1000], use_bias=True, activation=None))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Result of read_model for EfficientNet-b0 (from our counting code)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "mine = [('_conv_stem', Conv2D(input_size=224, kernel_shape=[3, 3, 3, 32], strides=(2, 2), padding='same', use_bias=True, activation='swish')), ('block_0', [('_depthwise_conv', DepthWiseConv2D(input_size=112, kernel_shape=[3, 3, 32, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=112, n_channels=32)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 32, 8], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 8, 32], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=112, n_channels=32)), ('_project_conv', Conv2D(input_size=112, kernel_shape=[1, 1, 32, 16], strides=(1, 1), padding='same', use_bias=True, activation=None))]), ('block_1', [('_expand_conv', Conv2D(input_size=112, kernel_shape=[1, 1, 16, 96], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=112, kernel_shape=[3, 3, 96, 1], strides=(2, 2), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=56, n_channels=96)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 96, 4], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 4, 96], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=56, n_channels=96)), ('_project_conv', Conv2D(input_size=56, kernel_shape=[1, 1, 96, 24], strides=(1, 1), padding='same', use_bias=True, activation=None))]), ('block_2', [('_expand_conv', Conv2D(input_size=56, kernel_shape=[1, 1, 24, 144], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=56, kernel_shape=[3, 3, 144, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=56, n_channels=144)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 144, 6], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 6, 144], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=56, n_channels=144)), ('_project_conv', Conv2D(input_size=56, kernel_shape=[1, 1, 144, 24], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=56, n_channels=144))]), ('block_3', [('_expand_conv', Conv2D(input_size=56, kernel_shape=[1, 1, 24, 144], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=56, kernel_shape=[5, 5, 144, 1], strides=(2, 2), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=28, n_channels=144)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 144, 6], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 6, 144], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=28, n_channels=144)), ('_project_conv', Conv2D(input_size=28, kernel_shape=[1, 1, 144, 40], strides=(1, 1), padding='same', use_bias=True, activation=None))]), ('block_4', [('_expand_conv', Conv2D(input_size=28, kernel_shape=[1, 1, 40, 240], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=28, kernel_shape=[5, 5, 240, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=28, n_channels=240)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 240, 10], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 10, 240], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=28, n_channels=240)), ('_project_conv', Conv2D(input_size=28, kernel_shape=[1, 1, 240, 40], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=28, n_channels=240))]), ('block_5', [('_expand_conv', Conv2D(input_size=28, kernel_shape=[1, 1, 40, 240], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=28, kernel_shape=[3, 3, 240, 1], strides=(2, 2), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=14, n_channels=240)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 240, 10], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 10, 240], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=14, n_channels=240)), ('_project_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 240, 80], strides=(1, 1), padding='same', use_bias=True, activation=None))]), ('block_6', [('_expand_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 80, 480], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=14, kernel_shape=[3, 3, 480, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=14, n_channels=480)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 480, 20], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 20, 480], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=14, n_channels=480)), ('_project_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 480, 80], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=14, n_channels=480))]), ('block_7', [('_expand_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 80, 480], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=14, kernel_shape=[3, 3, 480, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=14, n_channels=480)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 480, 20], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 20, 480], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=14, n_channels=480)), ('_project_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 480, 80], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=14, n_channels=480))]), ('block_8', [('_expand_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 80, 480], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=14, kernel_shape=[5, 5, 480, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=14, n_channels=480)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 480, 20], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 20, 480], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=14, n_channels=480)), ('_project_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 480, 112], strides=(1, 1), padding='same', use_bias=True, activation=None))]), ('block_9', [('_expand_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 112, 672], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=14, kernel_shape=[5, 5, 672, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=14, n_channels=672)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 672, 28], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 28, 672], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=14, n_channels=672)), ('_project_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 672, 112], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=14, n_channels=672))]), ('block_10', [('_expand_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 112, 672], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=14, kernel_shape=[5, 5, 672, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=14, n_channels=672)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 672, 28], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 28, 672], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=14, n_channels=672)), ('_project_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 672, 112], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=14, n_channels=672))]), ('block_11', [('_expand_conv', Conv2D(input_size=14, kernel_shape=[1, 1, 112, 672], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=14, kernel_shape=[5, 5, 672, 1], strides=(2, 2), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=7, n_channels=672)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 672, 28], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 28, 672], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=7, n_channels=672)), ('_project_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 672, 192], strides=(1, 1), padding='same', use_bias=True, activation=None))]), ('block_12', [('_expand_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 192, 1152], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=7, kernel_shape=[5, 5, 1152, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=7, n_channels=1152)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 1152, 48], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 48, 1152], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=7, n_channels=1152)), ('_project_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 1152, 192], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=7, n_channels=1152))]), ('block_13', [('_expand_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 192, 1152], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=7, kernel_shape=[5, 5, 1152, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=7, n_channels=1152)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 1152, 48], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 48, 1152], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=7, n_channels=1152)), ('_project_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 1152, 192], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=7, n_channels=1152))]), ('block_14', [('_expand_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 192, 1152], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=7, kernel_shape=[5, 5, 1152, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=7, n_channels=1152)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 1152, 48], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 48, 1152], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=7, n_channels=1152)), ('_project_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 1152, 192], strides=(1, 1), padding='same', use_bias=True, activation=None)), ('_skip_add', Add(input_size=7, n_channels=1152))]), ('block_15', [('_expand_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 192, 1152], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_depthwise_conv', DepthWiseConv2D(input_size=7, kernel_shape=[3, 3, 1152, 1], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_reduce_mean', GlobalAvg(input_size=7, n_channels=1152)), ('_se_reduce', Conv2D(input_size=1, kernel_shape=[1, 1, 1152, 48], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_se_expand', Conv2D(input_size=1, kernel_shape=[1, 1, 48, 1152], strides=(1, 1), padding='same', use_bias=True, activation='sigmoid')), ('_se_scale', Scale(input_size=7, n_channels=1152)), ('_project_conv', Conv2D(input_size=7, kernel_shape=[1, 1, 1152, 320], strides=(1, 1), padding='same', use_bias=True, activation=None))]), ('_conv_head', Conv2D(input_size=7, kernel_shape=[1, 1, 320, 1280], strides=(1, 1), padding='same', use_bias=True, activation='swish')), ('_avg_pooling', GlobalAvg(input_size=7, n_channels=1280)), ('_fc', FullyConnected(kernel_shape=[1280, 1000], use_bias=True, activation=None))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Verification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "official == mine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}