【multi precision】multi precision support (fp32 + fp16) #9339

xingjing1 · 2022-08-11T07:00:38Z

1.支持运行时混合精度
2.添加calib inplace算子
性能：

库体积影响：
+300KB

hong19860320 · 2022-08-23T06:47:18Z

cmake/configure.cmake

@@ -279,6 +279,7 @@ endif()
 if (LITE_ON_TINY_PUBLISH)
  add_definitions("-DLITE_ON_TINY_PUBLISH")
  add_definitions("-DLITE_ON_FLATBUFFERS_DESC_VIEW")
+  add_definitions("-DLITE_WITH_FLATBUFFERS_DESC")


可能影响模型加载耗时，需要评估下

目前测试的几个模型看来没有影响

shentanyue · 2022-08-23T08:30:04Z

lite/api/light_api.cc

@@ -194,7 +194,8 @@ void LightPredictor::PrepareFeedFetch() {
 }

 void LightPredictor::BuildRuntimeProgram(
-    const std::shared_ptr<const cpp::ProgramDesc>& program_desc) {
+    const std::shared_ptr<const cpp::ProgramDesc>& program_desc,
+    bool use_precision_low) {
  auto* exe_scope = &scope_->NewScope();


命名应该是 exec_scope

之前别人命名的，我这先不改了

shentanyue · 2022-08-23T08:32:12Z

lite/api/light_api.cc

+          if (op_type != "feed" && op_type != "fetch") {
+            if (place.precision == PRECISION(kFloat)) {
+              place.precision = PRECISION(kFP16);
+            } else if (place.precision == PRECISION(kAny)) {


判断可以合并
if (place.precision == PRECISION(kFloat) || place.precision == PRECISION(kAny)) {
function()...
}

shentanyue · 2022-08-23T08:38:12Z

lite/core/op_lite.cc

@@ -72,8 +72,8 @@ std::vector<std::unique_ptr<KernelBase>> OpLite::CreateKernels(
  auto pick_kernel = [&](const Place &place) {
    auto ks = KernelRegistry::Global().Create(
        op_type_, place.target, place.precision, place.layout);
-    VLOG(5) << "pick kernel for " << op_info()->Type() << " "
-            << place.DebugString() << " get " << ks.size() << " kernels";
+    // VLOG(5) << "pick kernel for " << op_info()->Type() << " "


可以清理下注释的代码

shentanyue · 2022-08-23T08:39:17Z

lite/core/op_lite.cc

@@ -130,6 +130,18 @@ bool OpLite::Attach(const cpp::OpDesc &opdesc, lite::Scope *scope) {
  return AttachImpl(*op_info(), scope);
 }

+#ifdef LITE_ON_FLATBUFFERS_DESC_VIEW
+bool OpLite::Attach(const cpp::OpDescWrite &opdesc, lite::Scope *scope) {
+  // valid_places_.clear();


同，注释的代码不用的话，可以清理下

shentanyue · 2022-08-23T08:42:50Z

lite/core/program.cc

+  int low_precision = 1;
+  std::string old_op;
+
+  if (use_precision_low == true) {


这里的逻辑直接改成这样更直观点？
use_precision_low_ = use_precision_low;
low_precision = use_precision_low_ ? 1 : 0;

shentanyue · 2022-08-23T08:44:27Z

lite/core/program.cc

+#ifdef ENABLE_ARM_FP16
+      if (lite::DeviceInfo::Global().has_fp16() && low_precision == 1) {
+        if (op_type != "feed" && op_type != "fetch") {
+          if (place.precision == static_cast<PrecisionType>(1)) {


直接用PrecisionType::kFloat这类枚举值，不要用1、5这类hard code，下同

ok，这地方忘改了

shentanyue · 2022-08-23T08:49:16Z

lite/core/program.cc

+            place.precision = static_cast<PrecisionType>(5);
+          }
+        }
+        // transfer weight to fp16


下面的代码是不是跟WeightFP32ToFP16这个函数重复了？

是的，我看看调用一下

很多地方不一样，应该调用不了

shentanyue · 2022-08-23T08:50:24Z

lite/core/program.cc

+        //  kernels = op->CreateKernels({place});
+        //}
+        if (kernels.size() == 0 && place.target == TargetType::kARM) {
+          place.target = TargetType::kHost;


这里是为什么要改成TargetType::kHost呢？

因为有些fp16的kernal是host下的

虽然也是arm的代码

hong19860320 · 2022-08-23T08:52:11Z

lite/api/paddle_api.h

@@ -559,6 +559,7 @@ class LITE_API MobileConfig : public ConfigBase {
  // whether to load data from memory. Model data will be loaded from memory
  // buffer if model_from_memory_ is true.
  bool model_from_memory_{false};
+  PrecisionMode pre_mode_{LITE_PRECISION_NORMAL};


不用简写成pre_mode吧，直接用 precision_mode_

shentanyue · 2022-08-23T08:58:12Z

lite/kernels/arm/calib_inplace_compute.h

+
+template <DataLayoutType DLType>
+class CalibComputeFp32ToInt32
+    : public KernelLite<TARGET(kARM), PRECISION(kInt32), DLType> {


PRECISION(kInt32)->PRECISION(kFloat)

ok，这里用不上，之后删掉

shentanyue · 2022-08-23T08:58:32Z

lite/kernels/arm/calib_inplace_compute.h

+
+template <DataLayoutType DLType>
+class CalibComputeFp32ToInt64
+    : public KernelLite<TARGET(kARM), PRECISION(kInt64), DLType> {


PRECISION(kInt64)->PRECISION(kFloat)

ok，这里用不上，之后删掉

shentanyue · 2022-08-23T09:00:30Z

lite/operators/calib_inplace_op.h

+#ifdef LITE_ON_FLATBUFFERS_DESC_VIEW
+  bool AttachImpl(const cpp::OpDescWrite &opdesc, lite::Scope *scope) override;
+#endif
+  void *getparam() { return &param_; }


zhupengyang

LGTM

xingjing1 added 3 commits August 5, 2022 02:05

fix bugs

f25409e

fix bugs

9664807

fix bugs

4df8c61

xingjing1 requested review from zhupengyang, hong19860320 and liyancas as code owners August 11, 2022 07:00

xingjing1 changed the title ~~Fix fp16 bugs~~ 【muti precision】fp16 and fp32 Aug 11, 2022

xingjing1 changed the title ~~【muti precision】fp16 and fp32~~ 【muti precision】muti precision support Aug 11, 2022

xingjing1 added 6 commits August 12, 2022 06:52

fix bugs

816cda1

fix bugs

683a21c

fix bugs

afbde41

fix bugs

a70815e

fix full_publish bugs

5a88ffb

fix bugd

4ee128d

hong19860320 reviewed Aug 23, 2022

View reviewed changes

xingjing1 added 2 commits August 23, 2022 07:22

fix bugs

c9c9dbb

fix bugs

abbec02

shentanyue reviewed Aug 23, 2022

View reviewed changes

hong19860320 reviewed Aug 23, 2022

View reviewed changes

shentanyue reviewed Aug 23, 2022

View reviewed changes

fix bugs

0f3da67

fix bugs

37f292f

zhupengyang approved these changes Sep 1, 2022

View reviewed changes

zhupengyang merged commit 5e35c5e into PaddlePaddle:develop Sep 1, 2022

zhupengyang changed the title ~~【muti precision】muti precision support~~ 【multi precision】multi precision support (fp32 + fp16) Sep 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

【multi precision】multi precision support (fp32 + fp16) #9339

【multi precision】multi precision support (fp32 + fp16) #9339

xingjing1 commented Aug 11, 2022 •

edited

Loading

hong19860320 Aug 23, 2022

xingjing1 Aug 23, 2022

shentanyue Aug 23, 2022

xingjing1 Aug 23, 2022

shentanyue Aug 23, 2022

xingjing1 Aug 23, 2022

shentanyue Aug 23, 2022

xingjing1 Aug 23, 2022

shentanyue Aug 23, 2022

xingjing1 Aug 23, 2022

shentanyue Aug 23, 2022

xingjing1 Aug 23, 2022

shentanyue Aug 23, 2022

xingjing1 Aug 23, 2022

shentanyue Aug 23, 2022

xingjing1 Aug 23, 2022

xingjing1 Aug 24, 2022

shentanyue Aug 23, 2022

xingjing1 Aug 23, 2022

xingjing1 Aug 23, 2022

hong19860320 Aug 23, 2022

xingjing1 Aug 23, 2022

shentanyue Aug 23, 2022

xingjing1 Aug 23, 2022

shentanyue Aug 23, 2022

xingjing1 Aug 23, 2022

shentanyue Aug 23, 2022

xingjing1 Aug 23, 2022

zhupengyang left a comment

【multi precision】multi precision support (fp32 + fp16) #9339

【multi precision】multi precision support (fp32 + fp16) #9339

Conversation

xingjing1 commented Aug 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhupengyang left a comment

Choose a reason for hiding this comment

xingjing1 commented Aug 11, 2022 •

edited

Loading