From 8155600bcf011e624c7ef60eaa8def54bcb931cf Mon Sep 17 00:00:00 2001 From: VoyagerXvoyagerx <19211416@bjtu.edu.cn> Date: Tue, 28 Feb 2023 17:04:46 +0800 Subject: [PATCH 01/23] [Doc] Translate model complexity analysis into Chinese. --- .../advanced_tutorials/model_analysis.md | 170 +++++++++++++++++- 1 file changed, 169 insertions(+), 1 deletion(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index 34a7a57ddf..62e34a09e6 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -1,3 +1,171 @@ # 模型复杂度分析 -翻译中，请暂时阅读英文文档 [Model Complexity Analysis](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/model_analysis.html)。 +我们提供了一个工具来帮助进行网络的复杂性分析。我们借鉴了 [fvcore](https://github.com/facebookresearch/fvcore) 的实现思路来构建这个工具，并计划在未来支持更多的自定义运算符。目前的工具提供了用于计算给定模型的 "parameter"、"activation" 和 "flops "的接口，并支持以网络结构或表格的形式逐层打印相关信息，同时提供operator层级和模块级的flop计数。如果您对如何准确测量一个 operator 的flop的实现细节感兴趣，请参考 [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md)。 + +## 什么是 FLOPs + +浮点运算数（FLOPs）在复杂性分析中不是一个定义非常明确的指标，我们按照 [detectron2](https://detectron2.readthedocs.io/en/latest/modules/fvcore.html#fvcore.nn.FlopCountAnalysis)，使用 1 组乘-加运算作为 1 个 flop。 + +## 什么是 activation + +激活量（activation）用于衡量某一层产生的特征数量。 + +例如，给定输入尺寸 `inputs = torch.randn((1, 3, 10, 10))`，和一个有3个输入通道、10个输出通道的线性层 `conv = nn.Conv2d(in_channels=3, out_channels=10, kernel_size=1)`。 + +将 `inputs` 输入到 `conv`，得到的输出特征图的尺寸是 `(1, 10, 10, 10)`。则 `output` 对这个 `conv` 层的激活量就是 `1000=10*10*10`。 + +让我们从下面的示例开始上手。 + +## 用法示例1: 从原始的 nn.Module 构建的模型 + +### 代码 + +```python +import torch +from torch import nn +from mmengine.analysis import get_model_complexity_info +# 以字典的形式返回分析结果，包括: +# ['flops', 'flops_str', 'activations', 'activations_str', 'params', 'params_str', 'out_table', 'out_arch'] + +class InnerNet(nn.Module): + def __init__(self): + super().__init__() + self.fc1 = nn.Linear(10,10) + self.fc2 = nn.Linear(10,10) + def forward(self, x): + return self.fc1(self.fc2(x)) + + +class TestNet(nn.Module): + def __init__(self): + super().__init__() + self.fc1 = nn.Linear(10,10) + self.fc2 = nn.Linear(10,10) + self.inner = InnerNet() + def forward(self, x): + return self.fc1(self.fc2(self.inner(x))) + +input_shape = (1, 10) +model = TestNet() + +analysis_results = get_model_complexity_info(model, input_shape) + +print(analysis_results['out_table']) +print(analysis_results['out_arch']) + +print("Model Flops:{}".format(analysis_results['flops_str'])) +print("Model Parameters:{}".format(analysis_results['params_str'])) +``` + +### 结果描述 + +返回的输出是一个包含以下7个键的字典: + +- `flops`: flop 的总数, e.g., 10000, 10000 +- `flops_str`: 格式化的字符串, e.g., 1.0G, 100M +- `params`: 全部参数的数量, e.g., 10000, 10000 +- `params_str`: 格式化的字符串, e.g., 1.0G, 100M +- `activations`: 激活量的总数, e.g., 10000, 10000 +- `activations_str`: 格式化的字符串, e.g., 1.0G, 100M +- `out_table`: 以表格形式打印相关信息 + +``` ++---------------------+----------------------+--------+--------------+ +| module | #parameters or shape | #flops | #activations | ++---------------------+----------------------+--------+--------------+ +| model | 0.44K | 0.4K | 40 | +| fc1 | 0.11K | 100 | 10 | +| fc1.weight | (10, 10) | | | +| fc1.bias | (10,) | | | +| fc2 | 0.11K | 100 | 10 | +| fc2.weight | (10, 10) | | | +| fc2.bias | (10,) | | | +| inner | 0.22K | 0.2K | 20 | +| inner.fc1 | 0.11K | 100 | 10 | +| inner.fc1.weight | (10, 10) | | | +| inner.fc1.bias | (10,) | | | +| inner.fc2 | 0.11K | 100 | 10 | +| inner.fc2.weight | (10, 10) | | | +| inner.fc2.bias | (10,) | | | ++---------------------+----------------------+--------+--------------+ +``` + +- `out_arch`: 以网络层级结构打印相关信息 + +```bash +TestNet( + #params: 0.44K, #flops: 0.4K, #acts: 40 + (fc1): Linear( + in_features=10, out_features=10, bias=True + #params: 0.11K, #flops: 100, #acts: 10 + ) + (fc2): Linear( + in_features=10, out_features=10, bias=True + #params: 0.11K, #flops: 100, #acts: 10 + ) + (inner): InnerNet( + #params: 0.22K, #flops: 0.2K, #acts: 20 + (fc1): Linear( + in_features=10, out_features=10, bias=True + #params: 0.11K, #flops: 100, #acts: 10 + ) + (fc2): Linear( + in_features=10, out_features=10, bias=True + #params: 0.11K, #flops: 100, #acts: 10 + ) + ) +) +``` + +## 用法示例2: 通过 mmengine 构建的模型 + +### 代码 + +```python +import torch.nn.functional as F +im +from mmengine.model import BaseModel +from mmengine.analysis import get_model_complexity_info + + +class MMResNet50(BaseModel): + def __init__(self): + super().__init__() + self.resnet = torchvision.models.resnet50() + + def forward(self, imgs, labels=None, mode='tensor'): + x = self.resnet(imgs) + if mode == 'loss': + return {'loss': F.cross_entropy(x, labels)} + elif mode == 'predict': + return x, labels + elif mode == 'tensor': + return x + + +input_shape = (3, 224, 224) +model = MMResNet50() + +analysis_results = get_model_complexity_info(model, input_shape) + + +print("Model Flops:{}".format(analysis_results['flops_str'])) +print("Model Parameters:{}".format(analysis_results['params_str'])) +``` + +### 输出 + +```bash +Model Flops:4.145G +Model Parameters:25.557M +``` + +## 接口 + +我们提供了更多的选项来支持自定义输出内容： + +- `model`: (nn.Module) 待分析的模型 +- `input_shape`: (tuple) 输入尺寸, e.g., (3, 224, 224) +- `inputs`: (optional: torch.Tensor), 如果传入该参数, `input_shape` 会被忽略 +- `show_table`: (bool) 是否以表格形式返回统计结果，默认：True +- `show_arch`: (bool) 是否以网络结构形式返回统计结果，默认：True From 9e0eedcb9be8772d88fbab80d97c56bc01b17e8f Mon Sep 17 00:00:00 2001 From: VoyagerXvoyagerx <19211416@bjtu.edu.cn> Date: Tue, 28 Feb 2023 17:12:15 +0800 Subject: [PATCH 02/23] [Doc] Translate model complexity analysis into Chinese. --- .../advanced_tutorials/model_analysis.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index 62e34a09e6..82a9d99533 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -1,6 +1,6 @@ # 模型复杂度分析 -我们提供了一个工具来帮助进行网络的复杂性分析。我们借鉴了 [fvcore](https://github.com/facebookresearch/fvcore) 的实现思路来构建这个工具，并计划在未来支持更多的自定义运算符。目前的工具提供了用于计算给定模型的 "parameter"、"activation" 和 "flops "的接口，并支持以网络结构或表格的形式逐层打印相关信息，同时提供operator层级和模块级的flop计数。如果您对如何准确测量一个 operator 的flop的实现细节感兴趣，请参考 [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md)。 +我们提供了一个工具来帮助进行网络的复杂性分析。我们借鉴了 [fvcore](https://github.com/facebookresearch/fvcore) 的实现思路来构建这个工具，并计划在未来支持更多的自定义运算符。目前的工具提供了用于计算给定模型的 "parameter"、"activation" 和 "flops "的接口，并支持以网络结构或表格的形式逐层打印相关信息，同时提供 operator 层级和模块级的 flop 计数。如果您对如何准确测量一个运算符的 flop 的实现细节感兴趣，请参考 [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md)。 ## 什么是 FLOPs @@ -61,12 +61,12 @@ print("Model Parameters:{}".format(analysis_results['params_str'])) 返回的输出是一个包含以下7个键的字典: -- `flops`: flop 的总数, e.g., 10000, 10000 -- `flops_str`: 格式化的字符串, e.g., 1.0G, 100M -- `params`: 全部参数的数量, e.g., 10000, 10000 -- `params_str`: 格式化的字符串, e.g., 1.0G, 100M -- `activations`: 激活量的总数, e.g., 10000, 10000 -- `activations_str`: 格式化的字符串, e.g., 1.0G, 100M +- `flops`: flop 的总数, 例如, 10000, 10000 +- `flops_str`: 格式化的字符串, 例如, 1.0G, 100M +- `params`: 全部参数的数量, 例如, 10000, 10000 +- `params_str`: 格式化的字符串, 例如, 1.0G, 100M +- `activations`: 激活量的总数, 例如, 10000, 10000 +- `activations_str`: 格式化的字符串, 例如, 1.0G, 100M - `out_table`: 以表格形式打印相关信息 ``` @@ -165,7 +165,7 @@ Model Parameters:25.557M 我们提供了更多的选项来支持自定义输出内容： - `model`: (nn.Module) 待分析的模型 -- `input_shape`: (tuple) 输入尺寸, e.g., (3, 224, 224) +- `input_shape`: (tuple) 输入尺寸，例如 (3, 224, 224) - `inputs`: (optional: torch.Tensor), 如果传入该参数, `input_shape` 会被忽略 -- `show_table`: (bool) 是否以表格形式返回统计结果，默认：True -- `show_arch`: (bool) 是否以网络结构形式返回统计结果，默认：True +- `show_table`: (bool) 是否以表格形式返回统计结果，默认值：True +- `show_arch`: (bool) 是否以网络结构形式返回统计结果，默认值：True From e50ac18e9269f0f76d9b10e856c89467d0ed2eaf Mon Sep 17 00:00:00 2001 From: VoyagerXvoyagerx <19211416@bjtu.edu.cn> Date: Tue, 28 Feb 2023 17:20:19 +0800 Subject: [PATCH 03/23] [Docs] fix the description of the interface --- docs/en/advanced_tutorials/model_analysis.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/advanced_tutorials/model_analysis.md b/docs/en/advanced_tutorials/model_analysis.md index 77d436765e..95582e806a 100644 --- a/docs/en/advanced_tutorials/model_analysis.md +++ b/docs/en/advanced_tutorials/model_analysis.md @@ -168,4 +168,4 @@ We provide more options to support custom output - `input_shape`: (tuple) the shape of the input, e.g., (3, 224, 224) - `inputs`: (optional: torch.Tensor), if given, `input_shape` will be ignored - `show_table`: (bool) whether return the statistics in the form of table, default: True -- `show_arch`: (bool) whether return the statistics in the form of table, default: True +- `show_arch`: (bool) whether return the statistics by network layers, default: True From 5904c8d93c4e26985e151c70bebc50b4787987f5 Mon Sep 17 00:00:00 2001 From: Yijie Zheng <67947949+VoyagerXvoyagerx@users.noreply.github.com> Date: Thu, 2 Mar 2023 10:45:00 +0800 Subject: [PATCH 04/23] update introduction Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com> --- docs/zh_cn/advanced_tutorials/model_analysis.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index 82a9d99533..81f33bd772 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -1,6 +1,6 @@ # 模型复杂度分析 -我们提供了一个工具来帮助进行网络的复杂性分析。我们借鉴了 [fvcore](https://github.com/facebookresearch/fvcore) 的实现思路来构建这个工具，并计划在未来支持更多的自定义运算符。目前的工具提供了用于计算给定模型的 "parameter"、"activation" 和 "flops "的接口，并支持以网络结构或表格的形式逐层打印相关信息，同时提供 operator 层级和模块级的 flop 计数。如果您对如何准确测量一个运算符的 flop 的实现细节感兴趣，请参考 [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md)。 +我们提供了一个工具来帮助进行网络的复杂性分析。我们借鉴了 [fvcore](https://github.com/facebookresearch/fvcore) 的实现思路来构建这个工具，并计划在未来支持更多的自定义运算符。目前的工具提供了用于计算给定模型的参数量（parameter）、激活量（activation） 和浮点运算量（FLOP）的接口，并支持以网络结构或表格的形式逐层打印相关信息，同时提供算子（operator）级和模块级（Module）的浮点运算量统计。如果您对如何准确测量一个运算符的 flop 的实现细节感兴趣，请参考 [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md)。 ## 什么是 FLOPs From df3913e43529a34709b20b84544d398703e8d680 Mon Sep 17 00:00:00 2001 From: Yijie Zheng <67947949+VoyagerXvoyagerx@users.noreply.github.com> Date: Thu, 2 Mar 2023 10:47:40 +0800 Subject: [PATCH 05/23] Update description of FLOPs Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com> --- docs/zh_cn/advanced_tutorials/model_analysis.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index 81f33bd772..a83af4fbd8 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -4,7 +4,7 @@ ## 什么是 FLOPs -浮点运算数（FLOPs）在复杂性分析中不是一个定义非常明确的指标，我们按照 [detectron2](https://detectron2.readthedocs.io/en/latest/modules/fvcore.html#fvcore.nn.FlopCountAnalysis)，使用 1 组乘-加运算作为 1 个 flop。 +浮点运算数（FLOPs）在复杂性分析中不是一个定义非常明确的指标。参考 [detectron2](https://detectron2.readthedocs.io/en/latest/modules/fvcore.html#fvcore.nn.FlopCountAnalysis) 的描述，将一组乘加运算定义为 1 个 flop。 ## 什么是 activation From daac393ed4a432f67eb18c3eb19602051e0dc8b4 Mon Sep 17 00:00:00 2001 From: Yijie Zheng <67947949+VoyagerXvoyagerx@users.noreply.github.com> Date: Thu, 2 Mar 2023 10:48:06 +0800 Subject: [PATCH 06/23] Update activation Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com> --- docs/zh_cn/advanced_tutorials/model_analysis.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index a83af4fbd8..df9eadd792 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -6,7 +6,7 @@ 浮点运算数（FLOPs）在复杂性分析中不是一个定义非常明确的指标。参考 [detectron2](https://detectron2.readthedocs.io/en/latest/modules/fvcore.html#fvcore.nn.FlopCountAnalysis) 的描述，将一组乘加运算定义为 1 个 flop。 -## 什么是 activation +## 什么是激活量（activation） 激活量（activation）用于衡量某一层产生的特征数量。 From d348822da28aa0ee959a358898f94d7708832e0c Mon Sep 17 00:00:00 2001 From: Yijie Zheng <67947949+VoyagerXvoyagerx@users.noreply.github.com> Date: Thu, 2 Mar 2023 10:48:57 +0800 Subject: [PATCH 07/23] Update model description Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com> --- docs/zh_cn/advanced_tutorials/model_analysis.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index df9eadd792..095455c49c 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -10,7 +10,7 @@ 激活量（activation）用于衡量某一层产生的特征数量。 -例如，给定输入尺寸 `inputs = torch.randn((1, 3, 10, 10))`，和一个有3个输入通道、10个输出通道的线性层 `conv = nn.Conv2d(in_channels=3, out_channels=10, kernel_size=1)`。 +例如，给定输入尺寸 `inputs = torch.randn((1, 3, 10, 10))`，和一个输入通道为 3、输出通道为 10 的线性层 `conv = nn.Conv2d(in_channels=3, out_channels=10, kernel_size=1)`。 将 `inputs` 输入到 `conv`，得到的输出特征图的尺寸是 `(1, 10, 10, 10)`。则 `output` 对这个 `conv` 层的激活量就是 `1000=10*10*10`。 From 04b1ee8128e79911b4a9eafdefe8f6b2a7e108c1 Mon Sep 17 00:00:00 2001 From: Yijie Zheng <67947949+VoyagerXvoyagerx@users.noreply.github.com> Date: Thu, 2 Mar 2023 10:49:54 +0800 Subject: [PATCH 08/23] Beautify code style Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com> --- .../advanced_tutorials/model_analysis.md | 22 ++++++++++--------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index 095455c49c..eff1353816 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -21,40 +21,42 @@ ### 代码 ```python -import torch from torch import nn + from mmengine.analysis import get_model_complexity_info + + # 以字典的形式返回分析结果，包括: # ['flops', 'flops_str', 'activations', 'activations_str', 'params', 'params_str', 'out_table', 'out_arch'] - class InnerNet(nn.Module): def __init__(self): super().__init__() - self.fc1 = nn.Linear(10,10) - self.fc2 = nn.Linear(10,10) + self.fc1 = nn.Linear(10, 10) + self.fc2 = nn.Linear(10, 10) + def forward(self, x): return self.fc1(self.fc2(x)) - + class TestNet(nn.Module): def __init__(self): super().__init__() - self.fc1 = nn.Linear(10,10) - self.fc2 = nn.Linear(10,10) + self.fc1 = nn.Linear(10, 10) + self.fc2 = nn.Linear(10, 10) self.inner = InnerNet() + def forward(self, x): return self.fc1(self.fc2(self.inner(x))) + input_shape = (1, 10) model = TestNet() - analysis_results = get_model_complexity_info(model, input_shape) - print(analysis_results['out_table']) print(analysis_results['out_arch']) - print("Model Flops:{}".format(analysis_results['flops_str'])) print("Model Parameters:{}".format(analysis_results['params_str'])) + ``` ### 结果描述 From cbf25dc697aa5232de299ba43a712c3da3859c0a Mon Sep 17 00:00:00 2001 From: Yijie Zheng <67947949+VoyagerXvoyagerx@users.noreply.github.com> Date: Thu, 2 Mar 2023 10:50:29 +0800 Subject: [PATCH 09/23] Modify examples Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com> --- docs/zh_cn/advanced_tutorials/model_analysis.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index eff1353816..9dad76b0c6 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -63,12 +63,12 @@ print("Model Parameters:{}".format(analysis_results['params_str'])) 返回的输出是一个包含以下7个键的字典: -- `flops`: flop 的总数, 例如, 10000, 10000 -- `flops_str`: 格式化的字符串, 例如, 1.0G, 100M -- `params`: 全部参数的数量, 例如, 10000, 10000 -- `params_str`: 格式化的字符串, 例如, 1.0G, 100M -- `activations`: 激活量的总数, 例如, 10000, 10000 -- `activations_str`: 格式化的字符串, 例如, 1.0G, 100M +- `flops`: flop 的总数, 例如, 1000, 1000000 +- `flops_str`: 格式化的字符串, 例如, 1.0G, 1.0M +- `params`: 全部参数的数量, 例如, 1000, 1000000 +- `params_str`: 格式化的字符串, 例如, 1.0K, 1M +- `activations`: 激活量的总数, 例如, 1000, 1000000 +- `activations_str`: 格式化的字符串, 例如, 1.0G, 1M - `out_table`: 以表格形式打印相关信息 ``` From 6d8e3e1417c69d4d254b08b04d1577458d552540 Mon Sep 17 00:00:00 2001 From: Yijie Zheng <67947949+VoyagerXvoyagerx@users.noreply.github.com> Date: Thu, 2 Mar 2023 10:51:06 +0800 Subject: [PATCH 10/23] Upadate output description Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com> --- docs/zh_cn/advanced_tutorials/model_analysis.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index 9dad76b0c6..5e3e363d8d 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -61,7 +61,7 @@ print("Model Parameters:{}".format(analysis_results['params_str'])) ### 结果描述 -返回的输出是一个包含以下7个键的字典: +返回的 `analysis_results` 是一个包含以下 7 个键的字典: - `flops`: flop 的总数, 例如, 1000, 1000000 - `flops_str`: 格式化的字符串, 例如, 1.0G, 1.0M From a16f7b02e671056c763c11db2479be3deba746ae Mon Sep 17 00:00:00 2001 From: Yijie Zheng <67947949+VoyagerXvoyagerx@users.noreply.github.com> Date: Thu, 2 Mar 2023 11:05:18 +0800 Subject: [PATCH 11/23] Update docs/zh_cn/advanced_tutorials/model_analysis.md Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com> --- docs/zh_cn/advanced_tutorials/model_analysis.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index 5e3e363d8d..f34142985e 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -16,7 +16,7 @@ 让我们从下面的示例开始上手。 -## 用法示例1: 从原始的 nn.Module 构建的模型 +## 用法示例1: 从原始的 `nn.Module` 构建的模型 ### 代码 From 2a28e513f5b9a7c5a22b2faa35310c3780900a08 Mon Sep 17 00:00:00 2001 From: VoyagerXvoyagerx <19211416@bjtu.edu.cn> Date: Thu, 2 Mar 2023 12:02:42 +0800 Subject: [PATCH 12/23] Replace FLOPs with flop; fix typo --- docs/en/advanced_tutorials/model_analysis.md | 7 ++++--- docs/zh_cn/advanced_tutorials/model_analysis.md | 12 ++++++------ 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/en/advanced_tutorials/model_analysis.md b/docs/en/advanced_tutorials/model_analysis.md index 95582e806a..556fcd0860 100644 --- a/docs/en/advanced_tutorials/model_analysis.md +++ b/docs/en/advanced_tutorials/model_analysis.md @@ -2,7 +2,7 @@ We provide a tool to help with the complexity analysis for the network. We borrow the idea from the implementation of [fvcore](https://github.com/facebookresearch/fvcore) to build this tool, and plan to support more custom operators in the future. Currently, it provides the interfaces to compute "parameter", "activation" and "flops" of the given model, and supports printing the related information layer-by-layer in terms of network structure or table. The analysis tool provides both operator-level and module-level flop counts simultaneously. Please refer to [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md) for implementation details of how to accurately measure the flops of one operator if interested. -## What's FLOPs +## What's Flop Flop is not a well-defined metric in complexity analysis, we follow [detectron2](https://detectron2.readthedocs.io/en/latest/modules/fvcore.html#fvcore.nn.FlopCountAnalysis) to use one fused multiple-add as one flop. @@ -21,9 +21,10 @@ Let's start with the following examples. ### Code ```python -import torch from torch import nn from mmengine.analysis import get_model_complexity_info + + # return a dict of analysis results, including: # ['flops', 'flops_str', 'activations', 'activations_str', 'params', 'params_str', 'out_table', 'out_arch'] @@ -69,7 +70,7 @@ The return outputs is dict, which contains the following keys: - `activations_str`: with formatted string, e.g., 1.0G, 100M - `out_table`: print related information by table -``` +```text +---------------------+----------------------+--------+--------------+ | module | #parameters or shape | #flops | #activations | +---------------------+----------------------+--------+--------------+ diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index f34142985e..9ac965e43e 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -1,10 +1,10 @@ # 模型复杂度分析 -我们提供了一个工具来帮助进行网络的复杂性分析。我们借鉴了 [fvcore](https://github.com/facebookresearch/fvcore) 的实现思路来构建这个工具，并计划在未来支持更多的自定义运算符。目前的工具提供了用于计算给定模型的参数量（parameter）、激活量（activation） 和浮点运算量（FLOP）的接口，并支持以网络结构或表格的形式逐层打印相关信息，同时提供算子（operator）级和模块级（Module）的浮点运算量统计。如果您对如何准确测量一个运算符的 flop 的实现细节感兴趣，请参考 [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md)。 +我们提供了一个工具来帮助进行网络的复杂性分析。我们借鉴了 [fvcore](https://github.com/facebookresearch/fvcore) 的实现思路来构建这个工具，并计划在未来支持更多的自定义运算符。目前的工具提供了用于计算给定模型的参数量（parameter）、激活量（activation） 和浮点运算量）的接口，并支持以网络结构或表格的形式逐层打印相关信息，同时提供算子（operator）级和模块级（Module）的浮点运算量统计。如果您对如何准确测量一个运算符的 flop 的实现细节感兴趣，请参考 [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md)。 -## 什么是 FLOPs +## 什么是浮点运算量（flop） -浮点运算数（FLOPs）在复杂性分析中不是一个定义非常明确的指标。参考 [detectron2](https://detectron2.readthedocs.io/en/latest/modules/fvcore.html#fvcore.nn.FlopCountAnalysis) 的描述，将一组乘加运算定义为 1 个 flop。 +浮点运算量（flop）在复杂性分析中不是一个定义非常明确的指标。参考 [detectron2](https://detectron2.readthedocs.io/en/latest/modules/fvcore.html#fvcore.nn.FlopCountAnalysis) 的描述，将一组乘加运算定义为 1 个 flop。 ## 什么是激活量（activation） @@ -71,7 +71,7 @@ print("Model Parameters:{}".format(analysis_results['params_str'])) - `activations_str`: 格式化的字符串, 例如, 1.0G, 1M - `out_table`: 以表格形式打印相关信息 -``` +```text +---------------------+----------------------+--------+--------------+ | module | #parameters or shape | #flops | #activations | +---------------------+----------------------+--------+--------------+ @@ -125,7 +125,7 @@ TestNet( ```python import torch.nn.functional as F -im +import torchvision from mmengine.model import BaseModel from mmengine.analysis import get_model_complexity_info @@ -164,7 +164,7 @@ Model Parameters:25.557M ## 接口 -我们提供了更多的选项来支持自定义输出内容： +除了上述基本用法，get_model_complexity_info 还能接受以下参数，输出定制化的统计结果： - `model`: (nn.Module) 待分析的模型 - `input_shape`: (tuple) 输入尺寸，例如 (3, 224, 224) From 368df7cbc1978803fc499590ea8b9849a14e0c92 Mon Sep 17 00:00:00 2001 From: VoyagerXvoyagerx <19211416@bjtu.edu.cn> Date: Thu, 2 Mar 2023 13:11:43 +0800 Subject: [PATCH 13/23] Fix typo --- docs/zh_cn/advanced_tutorials/model_analysis.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index 9ac965e43e..9f4b183cb5 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -164,7 +164,7 @@ Model Parameters:25.557M ## 接口 -除了上述基本用法，get_model_complexity_info 还能接受以下参数，输出定制化的统计结果： +除了上述基本用法，`get_model_complexity_info` 还能接受以下参数，输出定制化的统计结果： - `model`: (nn.Module) 待分析的模型 - `input_shape`: (tuple) 输入尺寸，例如 (3, 224, 224) From 8bb761c45408a30b667bcb929fee9b8acc572fb8 Mon Sep 17 00:00:00 2001 From: VoyagerXvoyagerx <19211416@bjtu.edu.cn> Date: Thu, 2 Mar 2023 13:25:30 +0800 Subject: [PATCH 14/23] fix lint error --- docs/zh_cn/advanced_tutorials/model_analysis.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index 9f4b183cb5..6a5b8ab0eb 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -36,7 +36,7 @@ class InnerNet(nn.Module): def forward(self, x): return self.fc1(self.fc2(x)) - + class TestNet(nn.Module): def __init__(self): From 4eb4f81979b8befca63b4ef761eb899636a8a0ff Mon Sep 17 00:00:00 2001 From: Yijie Zheng <67947949+VoyagerXvoyagerx@users.noreply.github.com> Date: Thu, 9 Mar 2023 17:18:10 +0800 Subject: [PATCH 15/23] Update docs/zh_cn/advanced_tutorials/model_analysis.md Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> --- docs/zh_cn/advanced_tutorials/model_analysis.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index 6a5b8ab0eb..031acefd09 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -16,7 +16,7 @@ 让我们从下面的示例开始上手。 -## 用法示例1: 从原始的 `nn.Module` 构建的模型 +## 用法示例1: 基于 `nn.Module` 构建的模型 ### 代码 From 9e92de19f0167447a054b3512851279b81c780fa Mon Sep 17 00:00:00 2001 From: Yijie Zheng <67947949+VoyagerXvoyagerx@users.noreply.github.com> Date: Thu, 9 Mar 2023 17:18:26 +0800 Subject: [PATCH 16/23] Update docs/zh_cn/advanced_tutorials/model_analysis.md Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> --- docs/zh_cn/advanced_tutorials/model_analysis.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index 031acefd09..dad5041aab 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -119,7 +119,7 @@ TestNet( ) ``` -## 用法示例2: 通过 mmengine 构建的模型 +## 用法示例2: 基于 BaseModel（来自 MMEngine）构建的模型 ### 代码 From 70668695415539bc15a5e4d3b056406887423072 Mon Sep 17 00:00:00 2001 From: Yijie Zheng <67947949+VoyagerXvoyagerx@users.noreply.github.com> Date: Thu, 9 Mar 2023 17:18:43 +0800 Subject: [PATCH 17/23] Update docs/zh_cn/advanced_tutorials/model_analysis.md Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> --- docs/zh_cn/advanced_tutorials/model_analysis.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index dad5041aab..5e58393ac1 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -10,7 +10,7 @@ 激活量（activation）用于衡量某一层产生的特征数量。 -例如，给定输入尺寸 `inputs = torch.randn((1, 3, 10, 10))`，和一个输入通道为 3、输出通道为 10 的线性层 `conv = nn.Conv2d(in_channels=3, out_channels=10, kernel_size=1)`。 +例如，给定输入尺寸 `inputs = torch.randn((1, 3, 10, 10))`，和一个输入通道为 3、输出通道为 10 的卷积层 `conv = nn.Conv2d(in_channels=3, out_channels=10, kernel_size=1)`。 将 `inputs` 输入到 `conv`，得到的输出特征图的尺寸是 `(1, 10, 10, 10)`。则 `output` 对这个 `conv` 层的激活量就是 `1000=10*10*10`。 From 16f138b7a74e82141757303ed4159b43f75dcbe1 Mon Sep 17 00:00:00 2001 From: Yijie Zheng <67947949+VoyagerXvoyagerx@users.noreply.github.com> Date: Thu, 9 Mar 2023 17:19:08 +0800 Subject: [PATCH 18/23] Update docs/zh_cn/advanced_tutorials/model_analysis.md Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> --- docs/zh_cn/advanced_tutorials/model_analysis.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index 5e58393ac1..f3fded93ee 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -12,7 +12,7 @@ 例如，给定输入尺寸 `inputs = torch.randn((1, 3, 10, 10))`，和一个输入通道为 3、输出通道为 10 的卷积层 `conv = nn.Conv2d(in_channels=3, out_channels=10, kernel_size=1)`。 -将 `inputs` 输入到 `conv`，得到的输出特征图的尺寸是 `(1, 10, 10, 10)`。则 `output` 对这个 `conv` 层的激活量就是 `1000=10*10*10`。 +将 `inputs` 输入到 `conv`，输出的特征图尺寸是 `(1, 10, 10, 10)`，则 `output` 对这个 `conv` 层的激活量就是 `1000=10*10*10`。 让我们从下面的示例开始上手。 From 136f3727e2d031e344745044ea0bb23f862e1a35 Mon Sep 17 00:00:00 2001 From: Yijie Zheng <67947949+VoyagerXvoyagerx@users.noreply.github.com> Date: Thu, 9 Mar 2023 17:19:18 +0800 Subject: [PATCH 19/23] Update docs/zh_cn/advanced_tutorials/model_analysis.md Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> --- docs/zh_cn/advanced_tutorials/model_analysis.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index f3fded93ee..0ec12c2483 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -61,7 +61,7 @@ print("Model Parameters:{}".format(analysis_results['params_str'])) ### 结果描述 -返回的 `analysis_results` 是一个包含以下 7 个键的字典: +返回的 `analysis_results` 是一个包含 7 个值的字典: - `flops`: flop 的总数, 例如, 1000, 1000000 - `flops_str`: 格式化的字符串, 例如, 1.0G, 1.0M From 0352da098deb9a183cfa9410d6f73401f615ae00 Mon Sep 17 00:00:00 2001 From: Yijie Zheng <67947949+VoyagerXvoyagerx@users.noreply.github.com> Date: Thu, 9 Mar 2023 17:32:30 +0800 Subject: [PATCH 20/23] Update docs/zh_cn/advanced_tutorials/model_analysis.md Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> --- docs/zh_cn/advanced_tutorials/model_analysis.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index 0ec12c2483..00c6889b79 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -1,6 +1,6 @@ # 模型复杂度分析 -我们提供了一个工具来帮助进行网络的复杂性分析。我们借鉴了 [fvcore](https://github.com/facebookresearch/fvcore) 的实现思路来构建这个工具，并计划在未来支持更多的自定义运算符。目前的工具提供了用于计算给定模型的参数量（parameter）、激活量（activation） 和浮点运算量）的接口，并支持以网络结构或表格的形式逐层打印相关信息，同时提供算子（operator）级和模块级（Module）的浮点运算量统计。如果您对如何准确测量一个运算符的 flop 的实现细节感兴趣，请参考 [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md)。 +我们提供了一个工具来帮助分析网络的复杂性。我们借鉴了 [fvcore](https://github.com/facebookresearch/fvcore) 的实现思路来构建这个工具，并计划在未来支持更多的自定义算子。目前的工具提供了用于计算给定模型的参数量（parameter）、激活量（activation）和浮点运算量的接口，并支持以网络结构或表格的形式逐层打印相关信息，同时提供了算子级别（operator）和模块级别（Module）的统计。如果您对如何统计运算符 flop 的实现细节感兴趣，请参考 [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md)。 ## 什么是浮点运算量（flop） From 692a95fc761f2c2675b7cb9df6631184235590bd Mon Sep 17 00:00:00 2001 From: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> Date: Mon, 13 Mar 2023 14:09:46 +0800 Subject: [PATCH 21/23] Update model_analysis.md --- docs/en/advanced_tutorials/model_analysis.md | 145 ++++++++++--------- 1 file changed, 78 insertions(+), 67 deletions(-) diff --git a/docs/en/advanced_tutorials/model_analysis.md b/docs/en/advanced_tutorials/model_analysis.md index 556fcd0860..4682f754d8 100644 --- a/docs/en/advanced_tutorials/model_analysis.md +++ b/docs/en/advanced_tutorials/model_analysis.md @@ -1,24 +1,30 @@ # Model Complexity Analysis -We provide a tool to help with the complexity analysis for the network. We borrow the idea from the implementation of [fvcore](https://github.com/facebookresearch/fvcore) to build this tool, and plan to support more custom operators in the future. Currently, it provides the interfaces to compute "parameter", "activation" and "flops" of the given model, and supports printing the related information layer-by-layer in terms of network structure or table. The analysis tool provides both operator-level and module-level flop counts simultaneously. Please refer to [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md) for implementation details of how to accurately measure the flops of one operator if interested. +We provide a tool to help with the complexity analysis for the network. We borrow the idea from the implementation of [fvcore](https://github.com/facebookresearch/fvcore) to build this tool, and plan to support more custom operators in the future. Currently, it provides the interfaces to compute "FLOPs", "Activations" and "Parameters", of the given model, and supports printing the related information layer-by-layer in terms of network structure or table. The analysis tool provides both operator-level and module-level flop counts simultaneously. Please refer to [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md) for implementation details of how to accurately measure the flops of one operator if interested. -## What's Flop +## Definition -Flop is not a well-defined metric in complexity analysis, we follow [detectron2](https://detectron2.readthedocs.io/en/latest/modules/fvcore.html#fvcore.nn.FlopCountAnalysis) to use one fused multiple-add as one flop. +The model complexity has three indicators, namely floating-point operations (FLOPs), activations, and parameters. Their definitions are as follows: -## What's Activation +- FLOPs -Activation is used to measure the feature quantity produced from one layer. + Floating-point operations (FLOPs) is not a clearly defined indicator. Here, we refer to the description in [detectron2](https://detectron2.readthedocs.io/en/latest/modules/fvcore.html#fvcore.nn.FlopCountAnalysis), which defines a set of multiply-accumulate operations as 1 FLOP. -For example, given the inputs with shape `inputs = torch.randn((1, 3, 10, 10))`, and one linear layer with `conv = nn.Conv2d(in_channels=3, out_channels=10, kernel_size=1)`. +- Activations -We get the `output` with shape `(1, 10, 10, 10)` after feeding the `inputs` into `conv`. The activation quantity of `output` of this `conv` layer is `1000=10*10*10` + Activation is used to measure the feature quantity produced from one layer. -Let's start with the following examples. +- Parameters -## Usage Example 1: Model built with native nn.Module + The parameter count of a model. -### Code +For example, given an input size of `inputs = torch.randn((1, 3, 10, 10))` and a convolutional layer `conv = nn.Conv2d(in_channels=3, out_channels=10, kernel_size=3)`, if the output feature map size is `(1, 10, 8, 8)`, then its FLOPs are `17280 = 10*8*8*3*3*3` (where 10*8*8 represents the output feature map size, and 3*3*3 represents the computation for each output), activations are `640 = 10*8*8`, and the parameter count is `280 = 3*10*3*3 + 10` (where 3*10*3\*3 represents the size of weights, and 10 represents the size of bias). + +## Usage + +### Model built with native nn.Module + +Build a model ```python from torch import nn @@ -50,17 +56,9 @@ input_shape = (1, 10) model = TestNet() analysis_results = get_model_complexity_info(model, input_shape) - -print(analysis_results['out_table']) -print(analysis_results['out_arch']) - -print("Model Flops:{}".format(analysis_results['flops_str'])) -print("Model Parameters:{}".format(analysis_results['params_str'])) ``` -### Description of Results - -The return outputs is dict, which contains the following keys: +The `analysis_results` returned by `get_model_complexity_info` is a dict, which contains the following keys: - `flops`: number of total flops, e.g., 10000, 10000 - `flops_str`: with formatted string, e.g., 1.0G, 100M @@ -70,42 +68,44 @@ The return outputs is dict, which contains the following keys: - `activations_str`: with formatted string, e.g., 1.0G, 100M - `out_table`: print related information by table -```text -+---------------------+----------------------+--------+--------------+ -| module | #parameters or shape | #flops | #activations | -+---------------------+----------------------+--------+--------------+ -| model | 0.44K | 0.4K | 40 | -| fc1 | 0.11K | 100 | 10 | -| fc1.weight | (10, 10) | | | -| fc1.bias | (10,) | | | -| fc2 | 0.11K | 100 | 10 | -| fc2.weight | (10, 10) | | | -| fc2.bias | (10,) | | | -| inner | 0.22K | 0.2K | 20 | -| inner.fc1 | 0.11K | 100 | 10 | -| inner.fc1.weight | (10, 10) | | | -| inner.fc1.bias | (10,) | | | -| inner.fc2 | 0.11K | 100 | 10 | -| inner.fc2.weight | (10, 10) | | | -| inner.fc2.bias | (10,) | | | -+---------------------+----------------------+--------+--------------+ -``` - -- `out_arch`: print related information by network layers - -```bash -TestNet( - #params: 0.44K, #flops: 0.4K, #acts: 40 - (fc1): Linear( - in_features=10, out_features=10, bias=True - #params: 0.11K, #flops: 100, #acts: 10 - ) - (fc2): Linear( - in_features=10, out_features=10, bias=True - #params: 0.11K, #flops: 100, #acts: 10 - ) - (inner): InnerNet( - #params: 0.22K, #flops: 0.2K, #acts: 20 +Print the results + +- print related information by table + + ```python + print(analysis_results['out_table']) + ``` + + ```text + +---------------------+----------------------+--------+--------------+ + | module | #parameters or shape | #flops | #activations | + +---------------------+----------------------+--------+--------------+ + | model | 0.44K | 0.4K | 40 | + | fc1 | 0.11K | 100 | 10 | + | fc1.weight | (10, 10) | | | + | fc1.bias | (10,) | | | + | fc2 | 0.11K | 100 | 10 | + | fc2.weight | (10, 10) | | | + | fc2.bias | (10,) | | | + | inner | 0.22K | 0.2K | 20 | + | inner.fc1 | 0.11K | 100 | 10 | + | inner.fc1.weight | (10, 10) | | | + | inner.fc1.bias | (10,) | | | + | inner.fc2 | 0.11K | 100 | 10 | + | inner.fc2.weight | (10, 10) | | | + | inner.fc2.bias | (10,) | | | + +---------------------+----------------------+--------+--------------+ + ``` + +- print related information by network layers + + ```python + print(analysis_results['out_arch']) + ``` + + ```bash + TestNet( + #params: 0.44K, #flops: 0.4K, #acts: 40 (fc1): Linear( in_features=10, out_features=10, bias=True #params: 0.11K, #flops: 100, #acts: 10 @@ -114,13 +114,30 @@ TestNet( in_features=10, out_features=10, bias=True #params: 0.11K, #flops: 100, #acts: 10 ) + (inner): InnerNet( + #params: 0.22K, #flops: 0.2K, #acts: 20 + (fc1): Linear( + in_features=10, out_features=10, bias=True + #params: 0.11K, #flops: 100, #acts: 10 + ) + (fc2): Linear( + in_features=10, out_features=10, bias=True + #params: 0.11K, #flops: 100, #acts: 10 + ) + ) ) -) -``` + ``` -## Usage Example 2: Model built with mmengine +- print results with formatted string -### Code + ```python + print("Model Flops:{}".format(analysis_results['flops_str'])) + # Model Flops:0.4K + print("Model Parameters:{}".format(analysis_results['params_str'])) + # Model Parameters:0.44K + ``` + +### Model built with mmengine ```python import torch.nn.functional as F @@ -149,16 +166,10 @@ model = MMResNet50() analysis_results = get_model_complexity_info(model, input_shape) - print("Model Flops:{}".format(analysis_results['flops_str'])) +# Model Flops:4.145G print("Model Parameters:{}".format(analysis_results['params_str'])) -``` - -### Output - -```bash -Model Flops:4.145G -Model Parameters:25.557M +# Model Parameters:25.557M ``` ## Interface From a8a726f2883e5b6245140d3bb48725b7bd5e7831 Mon Sep 17 00:00:00 2001 From: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> Date: Mon, 13 Mar 2023 14:10:38 +0800 Subject: [PATCH 22/23] Update model_analysis.md --- .../advanced_tutorials/model_analysis.md | 147 ++++++++++-------- 1 file changed, 79 insertions(+), 68 deletions(-) diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index 00c6889b79..706f523d02 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -1,24 +1,30 @@ # 模型复杂度分析 -我们提供了一个工具来帮助分析网络的复杂性。我们借鉴了 [fvcore](https://github.com/facebookresearch/fvcore) 的实现思路来构建这个工具，并计划在未来支持更多的自定义算子。目前的工具提供了用于计算给定模型的参数量（parameter）、激活量（activation）和浮点运算量的接口，并支持以网络结构或表格的形式逐层打印相关信息，同时提供了算子级别（operator）和模块级别（Module）的统计。如果您对如何统计运算符 flop 的实现细节感兴趣，请参考 [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md)。 +我们提供了一个工具来帮助分析网络的复杂性。我们借鉴了 [fvcore](https://github.com/facebookresearch/fvcore) 的实现思路来构建这个工具，并计划在未来支持更多的自定义算子。目前的工具提供了用于计算给定模型的浮点运算量（flop）、激活量（activation）和参数量（parameter）的接口，并支持以网络结构或表格的形式逐层打印相关信息，同时提供了算子级别（operator）和模块级别（Module）的统计。如果您对统计运算符 flop 的实现细节感兴趣，请参考 [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md)。 -## 什么是浮点运算量（flop） +## 定义 -浮点运算量（flop）在复杂性分析中不是一个定义非常明确的指标。参考 [detectron2](https://detectron2.readthedocs.io/en/latest/modules/fvcore.html#fvcore.nn.FlopCountAnalysis) 的描述，将一组乘加运算定义为 1 个 flop。 +模型复杂度有 3 个指标，分别是浮点运算量（flop）、激活量（activation）以及参数量（parameter），它们的定义如下： -## 什么是激活量（activation） +- 浮点运算量 -激活量（activation）用于衡量某一层产生的特征数量。 + 浮点运算量不是一个定义非常明确的指标，在这里参考 [detectron2](https://detectron2.readthedocs.io/en/latest/modules/fvcore.html#fvcore.nn.FlopCountAnalysis) 的描述，将一组乘加运算定义为 1 个 flop。 -例如，给定输入尺寸 `inputs = torch.randn((1, 3, 10, 10))`，和一个输入通道为 3、输出通道为 10 的卷积层 `conv = nn.Conv2d(in_channels=3, out_channels=10, kernel_size=1)`。 +- 激活量 -将 `inputs` 输入到 `conv`，输出的特征图尺寸是 `(1, 10, 10, 10)`，则 `output` 对这个 `conv` 层的激活量就是 `1000=10*10*10`。 + 激活量用于衡量某一层产生的特征数量。 -让我们从下面的示例开始上手。 +- 参数量 -## 用法示例1: 基于 `nn.Module` 构建的模型 + 模型的参数量。 -### 代码 +例如，给定输入尺寸 `inputs = torch.randn((1, 3, 10, 10))`，和一个卷积层 `conv = nn.Conv2d(in_channels=3, out_channels=10, kernel_size=3)`，那么它输出的特征图尺寸为 `(1, 10, 8, 8)`，则它的浮点运算量是 `17280 = 10*8*8*3*3*3`（10*8*8 表示输出的特征图大小、3*3*3 表示每一个输出需要的计算量）、激活量是 `640 = 10*8*8`、参数量是 `280 = 3*10*3*3 + 10`（3*10*3\*3 表示权重的尺寸、10 表示偏置值的尺寸）。 + +## 用法 + +### 基于 `nn.Module` 构建的模型 + +构建模型 ```python from torch import nn @@ -51,17 +57,9 @@ class TestNet(nn.Module): input_shape = (1, 10) model = TestNet() -analysis_results = get_model_complexity_info(model, input_shape) -print(analysis_results['out_table']) -print(analysis_results['out_arch']) -print("Model Flops:{}".format(analysis_results['flops_str'])) -print("Model Parameters:{}".format(analysis_results['params_str'])) - ``` -### 结果描述 - -返回的 `analysis_results` 是一个包含 7 个值的字典: +`get_model_complexity_info` 返回的 `analysis_results` 是一个包含 7 个值的字典: - `flops`: flop 的总数, 例如, 1000, 1000000 - `flops_str`: 格式化的字符串, 例如, 1.0G, 1.0M @@ -71,42 +69,44 @@ print("Model Parameters:{}".format(analysis_results['params_str'])) - `activations_str`: 格式化的字符串, 例如, 1.0G, 1M - `out_table`: 以表格形式打印相关信息 -```text -+---------------------+----------------------+--------+--------------+ -| module | #parameters or shape | #flops | #activations | -+---------------------+----------------------+--------+--------------+ -| model | 0.44K | 0.4K | 40 | -| fc1 | 0.11K | 100 | 10 | -| fc1.weight | (10, 10) | | | -| fc1.bias | (10,) | | | -| fc2 | 0.11K | 100 | 10 | -| fc2.weight | (10, 10) | | | -| fc2.bias | (10,) | | | -| inner | 0.22K | 0.2K | 20 | -| inner.fc1 | 0.11K | 100 | 10 | -| inner.fc1.weight | (10, 10) | | | -| inner.fc1.bias | (10,) | | | -| inner.fc2 | 0.11K | 100 | 10 | -| inner.fc2.weight | (10, 10) | | | -| inner.fc2.bias | (10,) | | | -+---------------------+----------------------+--------+--------------+ -``` - -- `out_arch`: 以网络层级结构打印相关信息 - -```bash -TestNet( - #params: 0.44K, #flops: 0.4K, #acts: 40 - (fc1): Linear( - in_features=10, out_features=10, bias=True - #params: 0.11K, #flops: 100, #acts: 10 - ) - (fc2): Linear( - in_features=10, out_features=10, bias=True - #params: 0.11K, #flops: 100, #acts: 10 - ) - (inner): InnerNet( - #params: 0.22K, #flops: 0.2K, #acts: 20 +打印结果 + +- 以表格形式打印相关信息 + + ```python + print(analysis_results['out_table']) + ``` + + ```text + +---------------------+----------------------+--------+--------------+ + | module | #parameters or shape | #flops | #activations | + +---------------------+----------------------+--------+--------------+ + | model | 0.44K | 0.4K | 40 | + | fc1 | 0.11K | 100 | 10 | + | fc1.weight | (10, 10) | | | + | fc1.bias | (10,) | | | + | fc2 | 0.11K | 100 | 10 | + | fc2.weight | (10, 10) | | | + | fc2.bias | (10,) | | | + | inner | 0.22K | 0.2K | 20 | + | inner.fc1 | 0.11K | 100 | 10 | + | inner.fc1.weight | (10, 10) | | | + | inner.fc1.bias | (10,) | | | + | inner.fc2 | 0.11K | 100 | 10 | + | inner.fc2.weight | (10, 10) | | | + | inner.fc2.bias | (10,) | | | + +---------------------+----------------------+--------+--------------+ + ``` + +- 以网络层级结构打印相关信息 + + ```python + print(analysis_results['out_arch']) + ``` + + ```bash + TestNet( + #params: 0.44K, #flops: 0.4K, #acts: 40 (fc1): Linear( in_features=10, out_features=10, bias=True #params: 0.11K, #flops: 100, #acts: 10 @@ -115,13 +115,30 @@ TestNet( in_features=10, out_features=10, bias=True #params: 0.11K, #flops: 100, #acts: 10 ) + (inner): InnerNet( + #params: 0.22K, #flops: 0.2K, #acts: 20 + (fc1): Linear( + in_features=10, out_features=10, bias=True + #params: 0.11K, #flops: 100, #acts: 10 + ) + (fc2): Linear( + in_features=10, out_features=10, bias=True + #params: 0.11K, #flops: 100, #acts: 10 + ) + ) ) -) -``` + ``` -## 用法示例2: 基于 BaseModel（来自 MMEngine）构建的模型 +- 以字符串的形式打印结果 -### 代码 + ```python + print("Model Flops:{}".format(analysis_results['flops_str'])) + # Model Flops:0.4K + print("Model Parameters:{}".format(analysis_results['params_str'])) + # Model Parameters:0.44K + ``` + +### 基于 BaseModel（来自 MMEngine）构建的模型 ```python import torch.nn.functional as F @@ -150,19 +167,13 @@ model = MMResNet50() analysis_results = get_model_complexity_info(model, input_shape) - print("Model Flops:{}".format(analysis_results['flops_str'])) +# Model Flops:4.145G print("Model Parameters:{}".format(analysis_results['params_str'])) +# Model Parameters:25.557M ``` -### 输出 - -```bash -Model Flops:4.145G -Model Parameters:25.557M -``` - -## 接口 +## 其他接口 除了上述基本用法，`get_model_complexity_info` 还能接受以下参数，输出定制化的统计结果： From 4b95ea2aaf58782168bdc89a2db4392f20bfe63b Mon Sep 17 00:00:00 2001 From: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> Date: Mon, 13 Mar 2023 14:29:38 +0800 Subject: [PATCH 23/23] Apply suggestions from code review Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com> --- docs/en/advanced_tutorials/model_analysis.md | 2 +- docs/zh_cn/advanced_tutorials/model_analysis.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/en/advanced_tutorials/model_analysis.md b/docs/en/advanced_tutorials/model_analysis.md index 4682f754d8..da19c06bc7 100644 --- a/docs/en/advanced_tutorials/model_analysis.md +++ b/docs/en/advanced_tutorials/model_analysis.md @@ -18,7 +18,7 @@ The model complexity has three indicators, namely floating-point operations (FLO The parameter count of a model. -For example, given an input size of `inputs = torch.randn((1, 3, 10, 10))` and a convolutional layer `conv = nn.Conv2d(in_channels=3, out_channels=10, kernel_size=3)`, if the output feature map size is `(1, 10, 8, 8)`, then its FLOPs are `17280 = 10*8*8*3*3*3` (where 10*8*8 represents the output feature map size, and 3*3*3 represents the computation for each output), activations are `640 = 10*8*8`, and the parameter count is `280 = 3*10*3*3 + 10` (where 3*10*3\*3 represents the size of weights, and 10 represents the size of bias). +For example, given an input size of `inputs = torch.randn((1, 3, 10, 10))` and a convolutional layer `conv = nn.Conv2d(in_channels=3, out_channels=10, kernel_size=3)`, if the output feature map size is `(1, 10, 8, 8)`, then its FLOPs are `17280 = 10*8*8*3*3*3` (where 10*8*8 represents the output feature map size, and 3*3*3 represents the computation for each output), activations are `640 = 10*8*8`, and the parameter count is `280 = 3*10*3*3 + 10` (where 3*10*3*3 represents the size of weights, and 10 represents the size of bias). ## Usage diff --git a/docs/zh_cn/advanced_tutorials/model_analysis.md b/docs/zh_cn/advanced_tutorials/model_analysis.md index 706f523d02..c3867933d1 100644 --- a/docs/zh_cn/advanced_tutorials/model_analysis.md +++ b/docs/zh_cn/advanced_tutorials/model_analysis.md @@ -1,10 +1,10 @@ # 模型复杂度分析 -我们提供了一个工具来帮助分析网络的复杂性。我们借鉴了 [fvcore](https://github.com/facebookresearch/fvcore) 的实现思路来构建这个工具，并计划在未来支持更多的自定义算子。目前的工具提供了用于计算给定模型的浮点运算量（flop）、激活量（activation）和参数量（parameter）的接口，并支持以网络结构或表格的形式逐层打印相关信息，同时提供了算子级别（operator）和模块级别（Module）的统计。如果您对统计运算符 flop 的实现细节感兴趣，请参考 [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md)。 +我们提供了一个工具来帮助分析网络的复杂性。我们借鉴了 [fvcore](https://github.com/facebookresearch/fvcore) 的实现思路来构建这个工具，并计划在未来支持更多的自定义算子。目前的工具提供了用于计算给定模型的浮点运算量（FLOPs）、激活量（Activations）和参数量（Parameters）的接口，并支持以网络结构或表格的形式逐层打印相关信息，同时提供了算子级别（operator）和模块级别（Module）的统计。如果您对统计浮点运算量的实现细节感兴趣，请参考 [Flop Count](https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md)。 ## 定义 -模型复杂度有 3 个指标，分别是浮点运算量（flop）、激活量（activation）以及参数量（parameter），它们的定义如下： +模型复杂度有 3 个指标，分别是浮点运算量（FLOPs）、激活量（Activations）以及参数量（Parameters），它们的定义如下： - 浮点运算量