eager_local_interpreter_with_infer_cache

[clackhan]

在Eager local op interpreter中使用infer cache，以提高infer device/stream/tesnor_meta的效率

[github-actions]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8619/

[github-actions]

Speed stats:

GPU Name: NVIDIA GeForce GTX 1080 

❌ OneFlow resnet50 time: 129.3ms (= 12932.2ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 142.8ms (= 14280.3ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.10 (= 142.8ms / 129.3ms)

OneFlow resnet50 time: 75.7ms (= 7569.1ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 86.8ms (= 8678.0ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.15 (= 86.8ms / 75.7ms)

OneFlow resnet50 time: 48.3ms (= 9650.6ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 62.5ms (= 12496.0ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.29 (= 62.5ms / 48.3ms)

OneFlow resnet50 time: 36.4ms (= 7270.6ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 45.2ms (= 9040.7ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.24 (= 45.2ms / 36.4ms)

OneFlow resnet50 time: 31.3ms (= 6257.2ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 50.3ms (= 10053.3ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.61 (= 50.3ms / 31.3ms)

OneFlow swin dataloader time: 0.275s (= 55.002s / 200, num_workers=1)
PyTorch swin dataloader time: 0.152s (= 30.347s / 200, num_workers=1)
Relative speed: 0.552 (= 0.152s / 0.275s)

OneFlow swin dataloader time: 0.071s (= 14.175s / 200, num_workers=4)
PyTorch swin dataloader time: 0.040s (= 8.094s / 200, num_workers=4)
Relative speed: 0.571 (= 0.040s / 0.071s)

OneFlow swin dataloader time: 0.039s (= 7.842s / 200, num_workers=8)
PyTorch swin dataloader time: 0.022s (= 4.322s / 200, num_workers=8)
Relative speed: 0.551 (= 0.022s / 0.039s)

❌ OneFlow resnet50 time: 145.1ms (= 14511.1ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 169.1ms (= 16908.8ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.17 (= 169.1ms / 145.1ms)

OneFlow resnet50 time: 94.7ms (= 9467.3ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 113.4ms (= 11338.2ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.20 (= 113.4ms / 94.7ms)

OneFlow resnet50 time: 66.9ms (= 13383.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 88.5ms (= 17693.2ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.32 (= 88.5ms / 66.9ms)

OneFlow resnet50 time: 55.7ms (= 11147.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 74.8ms (= 14950.7ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.34 (= 74.8ms / 55.7ms)

OneFlow resnet50 time: 47.9ms (= 9585.4ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 69.3ms (= 13858.8ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.45 (= 69.3ms / 47.9ms)

[github-actions]

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

[lixinqi]

Maybe<Symbol<Device>> RawGetDefaultCpuDevice() { return Device::New("cpu", 0); }

[github-actions]

Speed stats:

[github-actions]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8619/

[wyg1997]

import oneflow
#  import torch as oneflow
import time

vectors = oneflow.rand(690000, 1)
ones = oneflow.ones(1)

start_time = time.time()
for i in range(690000):
    if i % 10000 == 0:
        print(f"{i}: {time.time()-start_time}s")
        start_time = time.time()
    vectors[i] = ones

这个 PR 引入了新的问题，就是在指令多的时候速度会非常慢（具体原因我还不了解）。上面的例子每次打印越来越慢，而这个 PR 的上一个 PR （ 6ccedd3 ）是没有这个问题的。

[lixinqi]

这是一开始的耗时：

这是运行一段时间的耗时：

就是cache太大了。不但没能加速，反而成为负优化。

[lixinqi]

这应该是unordered_map不适合于cache的场景导致的。

[lixinqi]

这是一开始的耗时： 这是运行一段时间的耗时： 就是cache太大了。不但没能加速，反而成为负优化。

在俊丞帮助下，确定了是hash函数的问题，应该调用HashCombine而不是异或操作。

[lixinqi]

这应该是unordered_map不适合于cache的场景导致的。

主要问题不是unordered_map，而是其自定义的hash函数没有写好。 已有这个pr解决： #8932