Add dynamic/static learning rate doc for the api2.0 #2459

wawltor · 2020-08-22T17:36:58Z

Add dynamic/static learning rate doc for the api2.0.

Related with PR PaddlePaddle/Paddle#26550

wawltor · 2020-08-23T01:02:20Z

wawltor · 2020-08-23T01:05:02Z

wawltor · 2020-08-23T01:07:23Z

wawltor · 2020-08-23T01:09:53Z

wawltor · 2020-08-23T01:11:40Z

zhwesky2010

需要在英文里 doc/fluid/api/optimizer.rst 加一下对应英文的内容

zhwesky2010

LinearLrWarmup的中文文档缺乏

zhwesky2010 · 2020-08-23T11:24:26Z

doc/fluid/api_cn/optimizer_cn/lr_scheduler_cn/NoamLR_cn.rst

+
+.. math::
+
+    decayed\_learning\_rate = learning\_rate * d_{model}^{-0.5} * min(global\_steps^{-0.5}, global\_steps * warmup\_steps^{-1.5})


global_steps统一换成epoch吧，后面那几种只有epoch的概念了

zhwesky2010 · 2020-08-23T11:25:21Z

doc/fluid/api_cn/optimizer_cn/lr_scheduler_cn/PiecewiseLR_cn.rst

+.. code-block:: text
+
+    例如，设定的boundaries列表为[100, 200]，候选学习率常量列表values为[1.0, 0.5, 0.1]，则：
+    1、在当前训练步数global_step小于100步，学习率值为1.0。


当前训练的epoch数

zhwesky2010 · 2020-08-23T11:25:36Z

doc/fluid/api_cn/optimizer_cn/lr_scheduler_cn/PolynomiaLR_cn.rst

+
+.. math::
+
+    decay\_steps &= decay\_steps * math.ceil(\frac{global\_step}{decay\_steps})  \\


zhwesky2010 · 2020-08-23T11:25:46Z

doc/fluid/api_cn/optimizer_cn/lr_scheduler_cn/PolynomiaLR_cn.rst

+
+.. math::
+
+    global\_step &= min(global\_step, decay\_steps) \\


wawltor · 2020-08-23T11:47:06Z

LinearLrWarmup的中文文档缺乏

已加上

TCChenlong · 2020-08-24T12:09:36Z

doc/fluid/api_cn/optimizer_cn/lr_scheduler_cn/NaturalExpLR_cn.rst

+    - **learning_rate** （float） - 初始学习率，数据类型为Python float。
+    - **gamma** （float）：衰减率。
+    - **last_epoch** （int，可选）: 上一轮的轮数，重启训练时设置为上一轮的epoch数。默认值为 -1，则为初始学习率。
+    - **verbose** （bool，可选）：如果是 `True` ，则在每一轮更新时在标准输出 `stdout` 输出一条信息。


缺少Default 下同

TCChenlong · 2020-08-24T12:09:55Z

doc/fluid/api_cn/optimizer_cn/lr_scheduler_cn/PiecewiseLR_cn.rst

+
+参数
+:::::::::
+    - **boundaries** （lis）：指定衰减的步数边界。列表的数据元素为Python int类型。


lis -> list

TCChenlong · 2020-08-24T12:10:54Z

doc/fluid/api_cn/optimizer_cn/lr_scheduler_cn/NaturalExpLR_cn.rst

+-------------------------------
+
+
+.. py:class:: paddle.optimizer.lr_scheduler.NaturalExpLR(learning_rate, gama, last_epoch=-1, verbose=False)


gama -> gamma default 值英文里是0.1

必选参数

TCChenlong · 2020-08-24T12:11:38Z

doc/fluid/api_cn/optimizer_cn/lr_scheduler_cn/NaturalExpLR_cn.rst

+参数
+:::::::::
+    - **learning_rate** （float） - 初始学习率，数据类型为Python float。
+    - **gamma** （float）：衰减率。


是否为optional

必选参数

TCChenlong · 2020-08-24T12:12:37Z

doc/fluid/api_cn/optimizer_cn/lr_scheduler_cn/InverseTimeLR_cn.rst

+-------------------------------
+
+
+.. py:class:: paddle.optimizer.lr_scheduler.InverseTimeLR(learning_rate, gama, last_epoch=-1, verbose=False)


TCChenlong · 2020-08-24T12:28:28Z

doc/fluid/api_cn/optimizer_cn/lr_scheduler_cn/LambdaLR_cn.rst

+    paddle.disable_static()
+    x = np.random.uniform(-1, 1, [10, 10]).astype("float32")
+    linear = paddle.nn.Linear(10, 10)
+    scheduler = paddle.optimizer.StepLR(learning_rate=0.5, step_size=5, gamma=0.8, verbose=True)


应该是LambdaLR

TCChenlong · 2020-08-24T12:30:32Z

doc/fluid/api_cn/optimizer_cn/lr_scheduler_cn/ReduceLROnPlateauLR_cn.rst

+:::::::::
+    - **learning_rate** （float） - 初始学习率，数据类型为Python float。
+    - **mode** （str，可选）'min' 和 'max' 之一。通常情况下，为 'min' ，此时当 loss 停止下降时学习率将减小。默认：'min' 。 （注意：仅在特殊用法时，可以将其设置为 'max' ，此时判断逻辑相反， loss 停止上升学习率才减小）
+    - **fator** （float，可选） - 学习率衰减的比例。new_lr = origin_lr * factor，它是值小于1.0的float型数字，默认: 0.1。


TCChenlong · 2020-08-24T12:31:00Z

doc/fluid/api_cn/optimizer_cn/lr_scheduler_cn/ReduceLROnPlateauLR_cn.rst

+    - **threshold_mode** （str，可选）- 'rel' 和 'abs' 之一。在 'rel' 模式下， loss 最小变化的阈值是 last_loss * threshold ， 其中 last_loss 是 loss 在上个epoch的值。在 'abs' 模式下，loss 最小变化的阈值是 threshold 。 默认：'rel'。
+   - **cooldown** （int，可选）- 在学习速率每次减小之后，会进入时长为 ``cooldown`` 个 step 的冷静期。默认：0。
+   - **min_lr** （float，可选） - 最小的学习率。减小后的学习率最低下界限。默认：0。
+   - **epsilon** （float，可选）- 如果新旧学习率间的差异小于 eps ，则不会更新。默认值:1e-8。


小于epsilon

TCChenlong · 2020-08-24T12:33:30Z

doc/fluid/api_cn/optimizer_cn/lr_scheduler_cn/StepLR_cn.rst

+    paddle.disable_static()
+    x = np.random.uniform(-1, 1, [10, 10]).astype("float32")
+    linear = paddle.nn.Linear(10, 10)
+    scheduler = paddle.optimizer.MultiStepLR(learning_rate=0.5, milestones=[2, 4, 6], gamma=0.8, verbose=True)


TCChenlong · 2020-08-24T12:33:43Z

doc/fluid/api_cn/optimizer_cn/lr_scheduler_cn/StepLR_cn.rst

+        y = paddle.static.data(name='y', shape=[-1, 4, 5])
+        z = paddle.static.nn.fc(x, 100)
+        loss = paddle.mean(z)
+        scheduler = paddle.optimizer.MultiStepLR(learning_rate=0.5, milestones=[2, 4, 6], gamma=0.8, verbose=True)


TCChenlong

LGTM

CLAassistant · 2020-08-27T02:22:33Z

All committers have signed the CLA.

* Add autoShape() speed profiling * Update common.py * Create README.md * Update hubconf.py * cleanuip

joey12300 and others added 4 commits August 20, 2020 12:27

add lr doc

43e9451

Add the CosineAnnealingLR, LambdaLR, ReduceLROnPlateauLR

9a2ce66

fix InverseTimeLR, NaturalExpLR, NoamLR, PiecewiseLR, PolynomiaLR

a4ccd2e

add the cn lr_scheduler doc, test=develop

8f70a34

update the sample code for the lr_schedule, test=develop

1375128

zhwesky2010 reviewed Aug 23, 2020

View reviewed changes

zhwesky2010 changed the title ~~Add dynamic learning rate doc for the api2.0~~ Add dynamic/static learning rate doc for the api2.0 Aug 23, 2020

zhwesky2010 reviewed Aug 23, 2020

View reviewed changes

update the lr_scheduler to doc, test=develop

5a4aa48

zhwesky2010 mentioned this pull request Aug 23, 2020

[2.0API] Reconstruct all API related to LR Scheduler, unify dygraph and static PaddlePaddle/Paddle#26550

Merged

zhwesky2010 and others added 3 commits August 24, 2020 02:56

add index for en doc, test=develop

20dec72

update the code for the api 2.0, test=develop

e1ea81a

update the code for lr doc, test=develop

aacf834

TCChenlong reviewed Aug 24, 2020

View reviewed changes

wawltor added 3 commits August 24, 2020 12:54

udpate the doc for the lr_scheduler, test=develop

415da55

update the doc for paddle api 2.0, test=develop

3245eda

update the api for the lr_scheduler, test=develop

5ad3596

TCChenlong approved these changes Aug 26, 2020

View reviewed changes

update the user.name for the api,test=develop

f00b3b6

wawltor force-pushed the add_lr_doc branch from 7804ed6 to f00b3b6 Compare August 27, 2020 04:52

wawltor added 2 commits August 28, 2020 00:31

update the sample code for the lr, test=develop

7e2d820

update the sample code for the StepLR, MultiStepLR,test=develop

375b0ab

wawltor merged commit bd0935e into PaddlePaddle:develop Aug 28, 2020

RichardWooSJTU pushed a commit to RichardWooSJTU/docs that referenced this pull request Apr 8, 2022

Add autoShape() speed profiling (PaddlePaddle#2459)

569757e

* Add autoShape() speed profiling * Update common.py * Create README.md * Update hubconf.py * cleanuip


		.. math::

		decayed\_learning\_rate = learning\_rate * d_{model}^{-0.5} * min(global\_steps^{-0.5}, global\_steps * warmup\_steps^{-1.5})


		.. math::

		decay\_steps &= decay\_steps * math.ceil(\frac{global\_step}{decay\_steps}) \\

		-------------------------------


		.. py:class:: paddle.optimizer.lr_scheduler.NaturalExpLR(learning_rate, gama, last_epoch=-1, verbose=False)

Add dynamic/static learning rate doc for the api2.0 #2459

Add dynamic/static learning rate doc for the api2.0 #2459

Conversation

wawltor commented Aug 22, 2020 • edited by zhwesky2010 Loading

wawltor commented Aug 23, 2020

wawltor commented Aug 23, 2020

wawltor commented Aug 23, 2020

wawltor commented Aug 23, 2020

wawltor commented Aug 23, 2020

zhwesky2010 left a comment

Choose a reason for hiding this comment

zhwesky2010 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wawltor commented Aug 23, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TCChenlong Aug 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TCChenlong left a comment

Choose a reason for hiding this comment

CLAassistant commented Aug 27, 2020 • edited Loading

wawltor commented Aug 22, 2020 •

edited by zhwesky2010

Loading

TCChenlong Aug 24, 2020 •

edited

Loading

CLAassistant commented Aug 27, 2020 •

edited

Loading