Feat logical slice ops grad #8337

wyg1997 · 2022-05-31T07:59:49Z

接 #8313

支持 LogicalSliceAssign/LogicalSlice 后向

…nto feat-slice_ops_support_2d_sbp

…ort_2d_sbp

…c/oneflow into feat-slice_ops_support_2d_sbp

…nto feat-logical_slice_ops_support_all_sbp

github-actions · 2022-06-02T03:45:07Z

Speed stats:

GPU Name: NVIDIA GeForce GTX 1080 

❌ OneFlow resnet50 time: 130.0ms (= 13001.6ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 145.0ms (= 14499.6ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.12 (= 145.0ms / 130.0ms)

OneFlow resnet50 time: 76.2ms (= 7621.0ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 85.4ms (= 8543.2ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.12 (= 85.4ms / 76.2ms)

OneFlow resnet50 time: 52.9ms (= 10585.0ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 58.3ms (= 11661.2ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.10 (= 58.3ms / 52.9ms)

OneFlow resnet50 time: 39.5ms (= 7893.7ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 49.5ms (= 9907.0ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.26 (= 49.5ms / 39.5ms)

OneFlow resnet50 time: 33.7ms (= 6733.2ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 39.2ms (= 7846.7ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.17 (= 39.2ms / 33.7ms)

OneFlow swin dataloader time: 0.384s (= 76.796s / 200, num_workers=1)
PyTorch swin dataloader time: 0.151s (= 30.165s / 200, num_workers=1)
Relative speed: 0.393 (= 0.151s / 0.384s)

OneFlow swin dataloader time: 0.068s (= 13.694s / 200, num_workers=4)
PyTorch swin dataloader time: 0.043s (= 8.661s / 200, num_workers=4)
Relative speed: 0.633 (= 0.043s / 0.068s)

OneFlow swin dataloader time: 0.041s (= 8.232s / 200, num_workers=8)
PyTorch swin dataloader time: 0.023s (= 4.576s / 200, num_workers=8)
Relative speed: 0.556 (= 0.023s / 0.041s)

❌ OneFlow resnet50 time: 146.4ms (= 14643.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 174.7ms (= 17472.9ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.19 (= 174.7ms / 146.4ms)

OneFlow resnet50 time: 99.4ms (= 9938.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 112.4ms (= 11240.3ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.13 (= 112.4ms / 99.4ms)

OneFlow resnet50 time: 71.4ms (= 14284.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 93.2ms (= 18638.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.30 (= 93.2ms / 71.4ms)

OneFlow resnet50 time: 59.1ms (= 11819.6ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 74.4ms (= 14873.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.26 (= 74.4ms / 59.1ms)

OneFlow resnet50 time: 53.7ms (= 10735.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 73.6ms (= 14729.2ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.37 (= 73.6ms / 53.7ms)

github-actions · 2022-06-02T03:53:36Z

CI failed when running job: cpu-module. PR label automerge has been removed

wyg1997 · 2022-06-02T07:51:37Z

oneflow/core/autograd/gradient_funcs/logical_slice.cpp

+        zeros = JUST(functional::ConsistentConstant(ctx->value_shape, 0, out_grads[0]->dtype(),
+                                                    parallel_desc, *JUST(GetSbpList(nd_sbp))));


这里不能按 out_grads 的 nd_sbp 来初始化，因为 value_shape 的某个维度不一定可以 Split

github-actions · 2022-06-06T12:25:23Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8337/

wyg1997 and others added 30 commits May 25, 2022 10:12

feat(SliceOp): slice ops support 2d sbp

92e2a18

Merge branch 'master' into feat-slice_ops_support_2d_sbp

deaa871

fix(SliceOp): fix [B, P] 2d sbp bug

f0ee0ed

refine error message

1572353

Merge remote-tracking branch 'origin/feat-slice_ops_support_2d_sbp' i…

1e11873

…nto feat-slice_ops_support_2d_sbp

fix bug in parallel_num == 1

36c2093

add comment

c3f617b

add warning and format

64cb1c9

add NOLINT for boxing check

cca0ad9

Merge branch 'master' into feat-slice_ops_support_2d_sbp

61ebf3e

Merge branch 'master' into feat-slice_ops_support_2d_sbp

5a56356

Merge branch 'master' into feat-slice_ops_support_2d_sbp

38eb61c

Merge branch 'master' into feat-slice_ops_support_2d_sbp

d724598

Merge branch 'master' into feat-slice_ops_support_2d_sbp

0afbab7

feat(LogicalSliceOps): support all nd_sbp

f138436

feat(LogicalSlice): support nd_sbp

266ebb3

Merge remote-tracking branch 'origin/master' into feat-slice_ops_supp…

e8ca7d0

…ort_2d_sbp

add error message

797a6ca

fix(AutoTest): fix auto_test bug in module.parameter pass

4de5066

auto format by CI

3787d87

fix(LogicalSliceAssign): skip test when 1n1d

44e7230

Merge branch 'feat-slice_ops_support_2d_sbp' of github.com:Oneflow-In…

cbda49b

…c/oneflow into feat-slice_ops_support_2d_sbp

Merge remote-tracking branch 'origin/feat-slice_ops_support_2d_sbp' i…

4f9f1f3

…nto feat-logical_slice_ops_support_all_sbp

Merge branch 'master' into feat-logical_slice_ops_support_all_sbp

63abe10

fix SliceParams memset error

8284bb3

Merge branch 'master' into feat-logical_slice_ops_support_all_sbp

e20e423

remove memset

97f356f

add CHECK_JUST

9b2c4ff

fix(*): make sure split_axis >= 0 or equal to SPLIT_AXIS_FOR_NON_SPLIT

9f52bcb

Merge branch 'master' into feat-logical_slice_ops_support_all_sbp

8d0b8d0

wyg1997 added 2 commits June 2, 2022 10:54

modify as clang-tidy

78deade

Merge branch 'master' into feat-logical_slice_ops_grad

f780aea

wyg1997 added the automerge label Jun 2, 2022

wyg1997 requested review from oneflow-ci-bot and removed request for oneflow-ci-bot June 2, 2022 02:57

Merge branch 'master' into feat-logical_slice_ops_grad

6bb247e

github-actions bot removed the automerge label Jun 2, 2022

wyg1997 commented Jun 2, 2022

View reviewed changes

wyg1997 added 2 commits June 6, 2022 11:03

LogicalSlice ops grad use input nd_sbp

79a8a46

Merge branch 'master' into feat-logical_slice_ops_grad

5e2ad53

wyg1997 added the automerge label Jun 6, 2022

wyg1997 requested review from oneflow-ci-bot and removed request for oneflow-ci-bot June 6, 2022 03:18

Merge branch 'master' into feat-logical_slice_ops_grad

13c103b

wyg1997 requested review from oneflow-ci-bot and removed request for oneflow-ci-bot June 6, 2022 06:14

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot June 6, 2022 07:47

Merge branch 'master' into feat-logical_slice_ops_grad

1f1d8c3

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot June 6, 2022 12:04

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot June 6, 2022 16:24

mergify bot merged commit 83ed0ba into master Jun 6, 2022

mergify bot deleted the feat-logical_slice_ops_grad branch June 6, 2022 18:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat logical slice ops grad #8337

Feat logical slice ops grad #8337

wyg1997 commented May 31, 2022

github-actions bot commented Jun 2, 2022

github-actions bot commented Jun 2, 2022

wyg1997 Jun 2, 2022

wyg1997 Jun 6, 2022

github-actions bot commented Jun 6, 2022

		zeros = JUST(functional::ConsistentConstant(ctx->value_shape, 0, out_grads[0]->dtype(),
		parallel_desc, *JUST(GetSbpList(nd_sbp))));

Feat logical slice ops grad #8337

Feat logical slice ops grad #8337

Conversation

wyg1997 commented May 31, 2022

github-actions bot commented Jun 2, 2022

github-actions bot commented Jun 2, 2022

wyg1997 Jun 2, 2022

Choose a reason for hiding this comment

wyg1997 Jun 6, 2022

Choose a reason for hiding this comment

github-actions bot commented Jun 6, 2022