Code for optical flow estimation (Summer internship project @ 2019)

[katsuma-inoue-42]

Motivation and Context

Description

DL model for optical flow estimation

How do you use it?

[NOTE] Please run them on high end computer with GPU.

Build docker

make build

Run docker

docker run -it -v [PATH_TO_BLUEOIL_DIR]:/home/blueoil -v [PATH_TO_DATASET_DIR]:/home/blueoil/dataset --user $(id -u):$(id -g) [DOCKER_IMAGE_NAME]

Train model

CUDA_VISIBLE_DEVICES=[GPU_DEVICE_ID:optional] python lmnet/executor/train.py -c lmnet/configs/core/optical_flow_estimation/lm_flownet_3_q_quarter.py -i [EXPERIMENTAL_ID]

Convert checkpoint into pb_file and so/elf binaries

python script/binary_converter/build.py --device_id [GPU_DEVICE_ID:optional] saved/[EXPERIMENTAL_ID]/checkpoint/save.ckpt-[CHECKPOINT_ID]

(on your local PC) Download and run elf test

python script/binary_converter/test.py tmp/[EXPERIMENTAL_ID]

(on your local PC) Upload elf / so binaries to DE10-Nano board

python script/binary_converter/send.py tmp/[EXPERIMENTAL_ID] --host [IP_ADDRESS] --home_dir [PATH_TO_YOUR_WORKING_DIR]

(on your local PC) Run demonstration with so x86_avx binary file

python lmnet/lmnet/networks/optical_flow_estimation/demo_so.py ./tmp/[EXPERIMENTAL_ID].prj/lib_x86_avx.so --disable_full_screen 

Run demonstration with so arm/fpga binary file with socket connection

(SERVER: on GPU machine)

CUDA_VISIBLE_DEVICES=[GPU_DEVICE_ID] python lmnet/lmnet/networks/optical_flow_estimation/demo_server.py -i [EXPERIMENTAL_ID] --host [IP_ADDRESS] --restore_path ./saved/[EXPERIMENTAL_ID]/checkpoints/save.ckpt-[CHECKCPOINT_ID] --port [PORT]

(SERVER: on De10-Nano)

python demo_server_so.py ./lib_fpga.so --host [IP_ADDRESS]

(CLIENT: on your local PC)

 python lmnet/lmnet/networks/optical_flow_estimation/demo_client.py --host [IP_ADDRESS_OF_THE_SERVER] --port [PORT] --image_size [IMAGE_SIZE:default=[384, 512]] --disable_full_screen

Model structure

You can change the number of layers and input image size by specifying CONV_DEPTH and SLICE_STEP in configuration file.

lm_6_layers_continuous/quantized_full

lm_3_layers_continous/quantized_quarter

Screenshots (if appropriate):

Sample output (lm_3_c_quarter.py)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature / Optimization (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• My change requires a change to the documentation.
• I have updated the documentation accordingly.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 3 committers have signed the CLA.

❌ primenumber
❌ ki-lm
❌ OscarWang114

---

primenumber seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 3 committers have signed the CLA.

❌ primenumber
❌ OscarWang114
❌ ki-lm

---

primenumber seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}