[NNAdapter][TIM-VX] Add fill_like,split,slice and fix conv2d_transpose #8177

yingshengBD · 2021-12-31T12:10:20Z

test=develop test=huawei_ascend_npu

paddle-bot-old · 2021-12-31T12:10:25Z

Thanks for your contribution!

hong19860320 · 2022-01-03T08:19:34Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/slice.cc

+namespace verisilicon_timvx {
+
+int ConvertSlice(Converter* converter, hal::Operation* operation) {
+#if 1


#if 1 这种代码不要出现在 PR中

fixed in new commit

hong19860320 · 2022-01-03T08:24:52Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/utility.h

@@ -56,6 +56,12 @@ std::shared_ptr<tim::vx::Tensor> CreateTimVXTensor(
    const NNAdapterOperandType* type,
    void* buffer = nullptr,
    std::vector<int32_t> dimensions = {});
-
+std::shared_ptr<tim::vx::Tensor> CreateConstantTimVXTensor(


是否有必要加？除了converter会调用？还有哪里会使用？engine.cc ？

fixed in new commit, 删除该接口，现在使用addXXXoperand就不再需要这接口了

hong19860320 · 2022-01-03T08:28:39Z

lite/kernels/nnadapter/converter/conv2d_transpose.cc

+  }
+  NNAdapterOperand* filter_operand = nullptr;
+  bool is_quant_mode = false;
+  if (filter_precison == PRECISION(kInt8)) {


是否可以参考conv2d ?有些没有必要的代码是否能删掉？由于现在pengyang在 lite pass 已经增加混合精度的处理，所以这里理论上应该更少的类型判断才对

hong19860320 · 2022-01-03T08:33:35Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/conv2d_transpose.cc

@@ -25,6 +26,7 @@ int ConvertConv2DTranspose(Converter* converter, hal::Operation* operation) {
  CONV_2D_TRANSPOSE_OPERATION_EXTRACT_INPUTS_OUTPUTS
  // Dynamic shapes are still not supported
  NNADAPTER_CHECK_EQ(input_operand->type.dimensions.dynamic_count, 0);
+


为什么加这么多空行？如果有必要区分，建议加一行注释

为什么 // Convert to tim-vx tensors and operators 上面反而不空一行（用于分隔非硬件和硬件的代码）？

fixed in new commit

hong19860320 · 2022-01-03T08:35:06Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/conv2d_transpose.cc

  int32_t multiplier = 0;
  std::vector<int32_t> filter_dimensions(
      filter_operand->type.dimensions.data,
      filter_operand->type.dimensions.data +
          filter_operand->type.dimensions.count);
+


fixed in new commit

hong19860320 · 2022-01-03T08:35:12Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/conv2d_transpose.cc

  if (is_depthwise_mode) {
    multiplier = output_channel_size / group;
    NNADAPTER_CHECK_GT(filter_operand->type.dimensions.count, 2);
    // Oc,1,H,W -> 1,Oc,H,W
    filter_dimensions[0] = filter_dimensions[1];
    filter_dimensions[1] = output_channel_size;
  }
+


fixed in new commit

hong19860320 · 2022-01-03T08:55:29Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/conv2d_transpose.cc

@@ -48,18 +50,24 @@ int ConvertConv2DTranspose(Converter* converter, hal::Operation* operation) {
  if (!input_tensor) {
    input_tensor = converter->ConvertOperand(input_operand);
  }
+
+  std::vector<int32_t> filter_permutation = {1, 0, 2, 3};


为什么需要执行 oc, ic, h, w -> ic,oc,h,w? TIM-VX 默认是WHIcOc的啊 https://github.com/VeriSilicon/TIM-VX/blob/eecbe264b6a934ad6b089aa4701fa3fa77335201/include/tim/vx/ops/deconv.h#L63

正常conv是Oc为filter的batch数，而deconv中的Oc正是filter的channel数，所以这里需要transpose一下，这个修改经过单测和模型验证

hong19860320 · 2022-01-03T08:58:04Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/conv2d_transpose.cc

  int32_t multiplier = 0;
  std::vector<int32_t> filter_dimensions(
      filter_operand->type.dimensions.data,
      filter_operand->type.dimensions.data +
          filter_operand->type.dimensions.count);
+
  if (is_depthwise_mode) {


deconv存在depthwise的情况吗？如果不存在就不要做如下处理

fixed in new commit

hong19860320 · 2022-01-03T09:01:13Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/converter.h

@@ -48,6 +48,13 @@ class Converter {
  // operand
  std::shared_ptr<tim::vx::Tensor> ConvertOperand(
      hal::Operand* operand, std::vector<int32_t> dimensions = {});
+  // Add a constant TIM-VX tensor


为什么不加在AddTensor下面？

fixed in new commit, 删除该接口，现在使用addXXXoperand就不再需要这接口了

hong19860320 · 2022-01-03T09:02:01Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/converter.cc

@@ -94,5 +94,16 @@ std::shared_ptr<tim::vx::Tensor> Converter::ConvertOperand(
  return tensor;
 }

+std::shared_ptr<tim::vx::Tensor> Converter::AddConstantTensor(


为什么不加在AddTensor下面？

fixed in new commit, 删除该接口，现在使用addXXXoperand就不再需要这接口了

hong19860320 · 2022-01-03T09:02:16Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/converter.cc

+    tim::vx::DataType precision,
+    const float* quant_scale,
+    const int32_t* zero_point) {
+  auto tensor = CreateConstantTimVXTensor(


为什么不直接调用CreateTimVXTensor

fixed in new commit, 删除该接口，现在使用addXXXoperand就不再需要这接口了

hong19860320 · 2022-01-03T09:04:06Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/fill_like.cc

+  auto output_tensor = converter->ConvertOperand(output_operand);
+  std::vector<int32_t> dimensions = {1};
+  std::shared_ptr<tim::vx::Tensor> dummy_tesnor;
+  if (IsUInt8AsymmPerLayerQuantType(input_operand->type.precision)) {


按照之前讨论的加pass的方法改一下，这个代码太丑了

在新提交代码中engine.cc的build model阶段，新增函数调用ConvertFillLikeIntoEltwiseMul，在进入driver hal的convert前就替换掉fill like算子

hong19860320 · 2022-01-03T09:05:06Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/slice.cc

+    input_tensor = converter->ConvertOperand(input_operand);
+  }
+  auto output_tensor = converter->ConvertOperand(output_operand);
+


不要无缘无故加空行，如果需要隔开，建议加一行注释

fixed in new commit

hong19860320 · 2022-01-03T09:05:59Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/slice.cc

+  }
+  auto output_tensor = converter->ConvertOperand(output_operand);
+
+  auto output_dim = output_operand->type.dimensions.count;


output_rank或者 output_dimensions_count

fixed in new commit

hong19860320 · 2022-01-03T09:07:04Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/slice.cc

+  auto output_tensor = converter->ConvertOperand(output_operand);
+
+  auto output_dim = output_operand->type.dimensions.count;
+  std::vector<int32_t> axis;


建议用mapped_axes

fixed in new commit

hong19860320 · 2022-01-03T09:13:01Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/slice.cc

+  for (uint32_t i = 0; i < axes_count; i++) {
+    axis.push_back(output_dim - 1 - axes[i]);
+  }
+  std::vector<int32_t> start;


建议用mapped_starts，变量取名不要随心所欲

fixed in new commit

hong19860320 · 2022-01-03T09:20:30Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/slice.cc

+    else
+      start.push_back(0);
+  }
+  std::vector<int32_t> length;


mapped_lengths

fixed in new commit

hong19860320 · 2022-01-03T09:21:07Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/slice.cc

+        break;
+      }
+    }
+    if (match_axis)


为什么不用 ? :

fixed in new commit

hong19860320 · 2022-01-03T09:21:24Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/slice.cc

+        break;
+      }
+    }
+    if (match_axis)


为什么不用 ? :

fixed in new commit

hong19860320 · 2022-01-03T09:22:16Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/slice.cc

+  }
+  std::vector<int32_t> start;
+  for (int i = 0; i < output_dim; i++) {
+    bool match_axis = false;


直接用found或者matched就行了

fixed in new commit

hong19860320 · 2022-01-03T09:24:37Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/split.cc

+    split_timvx.push_back(split[i]);
+    output_tensors[i] = converter->ConvertOperand(output_operands[i]);
+  }
+


建议用加一行注释代替空行

fixed in new commit

hong19860320 · 2022-01-03T09:25:07Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/split.cc

+  std::vector<std::shared_ptr<tim::vx::Tensor>> output_tensors(split.size());
+  // WHCN
+  uint32_t axis_timvx = input_operand->type.dimensions.count - 1 - axis;
+  std::vector<uint32_t> split_timvx;


mapped_split

fixed in new commit

hong19860320 · 2022-01-03T09:25:19Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/split.cc

+  }
+  std::vector<std::shared_ptr<tim::vx::Tensor>> output_tensors(split.size());
+  // WHCN
+  uint32_t axis_timvx = input_operand->type.dimensions.count - 1 - axis;


mapped_axis

fixed in new commit

hong19860320 · 2022-01-03T09:25:37Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/split.cc

+  }
+  std::vector<std::shared_ptr<tim::vx::Tensor>> output_tensors(split.size());
+  // WHCN
+  uint32_t axis_timvx = input_operand->type.dimensions.count - 1 - axis;


是否需要判断axis的正负号？

这里不需要，在前面SPLIT_OPERATION_EXTRACT_INPUTS_OUTPUTS中已经处理了axis为负数的情况并转为非负数

hong19860320 · 2022-01-03T09:29:36Z

lite/kernels/nnadapter/converter/conv2d_transpose.cc

@@ -130,12 +236,6 @@ int ConvertConv2dTranspose(Converter* converter, OpInfo* op, Scope* scope) {
  auto fuse_code_operand = converter->AddConstantOperand(fuse_code_value);

  // Output operand
-  auto output_name = op->Output("Output").front();


单测都补上

fixed in new commit

hong19860320 · 2022-01-03T09:31:55Z

问题太多了，先改一遍我再看看吧@yingshengBD

hong19860320 · 2022-01-05T11:34:42Z

...nnadapter/nnadapter/driver/verisilicon_timvx/optimizer/convert_fill_like_into_eltwise_mul.cc

+      RemoveOperand(model, value_operand);
+      hal::Operand* dummy_operand;
+      const std::vector<int32_t> dimensions({1});
+      if (IsUInt8AsymmPerLayerQuantType(input_operand->type.precision)) {


应该是这个公式吧 input_operand * zero_operand + value_operand，zero_operand 可以用

Paddle-Lite/lite/backends/nnadapter/nnadapter/utility/modeling.cc

Line 579 in 07bfd07

auto zero_operand = AddOperand(model);

fixed in new commit，新增fill any like 单测

hong19860320 · 2022-01-12T07:48:46Z

lite/backends/nnadapter/nnadapter/driver/verisilicon_timvx/converter/conv2d_transpose.cc

  // Convert to tim-vx tensors and operators
  auto input_tensor = converter->GetMappedTensor(input_operand);
  if (!input_tensor) {
    input_tensor = converter->ConvertOperand(input_operand);
  }
+


以后需要注意下，不建议直接加一个空行，如果需要隔开，可以加一行注释

hong19860320 · 2022-01-12T07:50:52Z

...nds/nnadapter/nnadapter/driver/verisilicon_timvx/optimizer/convert_fill_like_into_mul_add.cc

+#include "utility/utility.h"
+
+namespace nnadapter {
+static void InsertAdd(hal::Model* model,


先这样吧，后面我来改这部分的代码

test=develop test=huawei_ascend_npu

test=develop test=huawei_ascend_npu test=verisilicon_timvx

hong19860320

LGTM

PaddlePaddle#8177)

hong19860320 reviewed Jan 3, 2022

View reviewed changes

hong19860320 changed the title ~~[NNAdapter][TIM-VX]add tim-vx OP: fill_like,split,slice;fix tim-vx OP:conv2d_transpose~~ [NNAdapter][TIM-VX]add fill_like,split,slice and fix conv2d_transpose Jan 3, 2022

hong19860320 changed the title ~~[NNAdapter][TIM-VX]add fill_like,split,slice and fix conv2d_transpose~~ [NNAdapter][TIM-VX] Add fill_like,split,slice and fix conv2d_transpose Jan 3, 2022

hong19860320 reviewed Jan 5, 2022

View reviewed changes

hong19860320 reviewed Jan 12, 2022

View reviewed changes

yingshengBD added 7 commits January 13, 2022 10:12

add tim-vx OP: fill_like,split,slice;fix tim-vx OP:conv2d_transpose

e3dc06d

test=develop test=huawei_ascend_npu

add tim-vx OP, fix review-comment

2ba230a

test=develop test=huawei_ascend_npu

fix review-comment,add fill_like u-test

13197f5

test=develop test=huawei_ascend_npu

delete redundant quant process in conv2d,conv2d_transpose,fc

9d20f86

test=develop test=huawei_ascend_npu

delete redundant quant process in conv2d,conv2d_transpose,fc

f23a9ed

test=develop test=huawei_ascend_npu test=verisilicon_timvx

fix conflict

f708771

test=develop test=huawei_ascend_npu test=verisilicon_timvx

TIMVX-main change the hard-sigmoid api

ec056bb

test=develop test=huawei_ascend_npu test=verisilicon_timvx

hong19860320 approved these changes Jan 15, 2022

View reviewed changes

hong19860320 merged commit 7dfbd5a into PaddlePaddle:develop Jan 15, 2022

WeiLi233 pushed a commit to WeiLi233/Paddle-Lite that referenced this pull request Mar 29, 2022

[NNAdapter][TIM-VX] Add fill_like,split,slice and fix conv2d_transpose (

d28bd05

PaddlePaddle#8177)

[NNAdapter][TIM-VX] Add fill_like,split,slice and fix conv2d_transpose #8177

[NNAdapter][TIM-VX] Add fill_like,split,slice and fix conv2d_transpose #8177

Conversation

yingshengBD commented Dec 31, 2021

paddle-bot-old bot commented Dec 31, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hong19860320 commented Jan 3, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hong19860320 Jan 12, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hong19860320 left a comment

Choose a reason for hiding this comment

hong19860320 Jan 12, 2022 •

edited

Loading