Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks

This repository contains the source code associated with Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks, ECCV 2020. This code has most recently been tested with Python 3.7 and Pytorch 1.1.0.

Introduction

Event-based cameras display great potential for a variety of tasks such as high-speed motion detection and navigation in low-light environments where conventional frame-based cameras suffer critically. This is attributed to their high temporal resolution, high dynamic range, and low-power consumption. However, conventional computer vision methods as well as deep Analog Neural Networks (ANNs) are not suited to work well with the asynchronous and discrete nature of event camera outputs. Spiking Neural Networks (SNNs) serve as ideal paradigms to handle event camera outputs, but deep SNNs suffer in terms of performance due to spike vanishing phenomenon. To overcome these issues, we present Spike-FlowNet, a deep hybrid neural network architecture integrating SNNs and ANNs for efficiently estimating optical flow from sparse event camera outputs without sacrificing the performance. The network is end-to-end trained with self-supervised learning on Multi-Vehicle Stereo Event Camera (MVSEC) dataset. Spike-FlowNet outperforms its corresponding ANN-based method in terms of the optical flow prediction capability while providing significant computational efficiency.

Installation

Clone this repository using: git clone https://github.com/chan8972/Spike-FlowNet.git

Create a conda environment using the environment.yml file: conda env create -f environment.yml

Activate the conda environment: conda activate spikeflownet

Dataset

The data for the outdoor_day and indoor_flying sequences can be found here.

Ground truth flow computed from the paper can also be downloaded here.

Download the *_data.hdf5 and *_gt.hdf5 files from the above link in their respective folders inside the /datasets.

Example: Download indoor_flying1_data.hdf5 and indoor_flying1_gt.hdf5 files into /datasets/indoor_flying1 folder.

Convert the hdf5 files into encoded format using /encoding/split_coding.py.

Training

The basic syntax is:

For dt=1 : python3 main_spike_flow_dt1.py

For dt=4 : python3 main_spike_flow_dt4.py

Pre-trained Model

The pretrained models for dt=1 and dt=4 can be found in /pretrain folder. Note, pretrained models are trained only on outdoor_day2 dataset as described in the paper.

Testing with flow visualization

The basic syntax is:

For dt=1 : python3 main_spike_flow_dt1.py --evaluate --pretrained='checkpoint_path' --render

For dt=4 : python3 main_spike_flow_dt4.py --evaluate --pretrained='checkpoint_path' --render

Some useful Command-line arguments

--data : specifies the dataset folder /datasets

--savedir : folder for saving training results

--workers : number of workers to use

--render : render flow outputs while evaluating

--evaluate-interval : how many epochs to evaluate after

--pretrained : path to pretrained model

Other available command line arguments for hyperparameter tuning can be found in the main_spike_flow_dt*.py files.

Citations

If you find this code useful in your research, please consider citing: Lee C., Kosta A.K., Zhu A.Z., Chaney K., Daniilidis K., Roy K. "Spike-FlowNet: Event-Based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks.", In Proceedings of the European Conference on Computer Vision (ECCV) 2020

@inproceedings{10.1007/978-3-030-58526-6_22,
  title={Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks},
  author={Lee, Chankyu and Kosta, Adarsh and Zhu, Alex Zihao and Chaney, Kenneth and Daniilidis, Kostas and Roy, Kaushik},
  booktitle={European Conference on Computer Vision},
  pages={366--382},
  year={2020},
  organization={Springer}
}

An arXiv version is also available: Chankyu Lee, Adarsh Kumar Kosta, Alex Zihao Zhu, Kenneth Chaney, Kostas Daniilidis and Kaushik Roy, "Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks", arXiv preprint, 2020

@article{lee2020spike,
  title={Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks},
  author={Lee, Chankyu and Kosta, Adarsh and Zhu, Alex Zihao and Chaney, Kenneth and Daniilidis, Kostas and Roy, Kaushik},
  journal={arXiv preprint arXiv:2003.06696},
  year={2020}
}

Authors

Chankyu Lee, Adarsh Kumar Kosta, Alex Zihao Zhu, Kenneth Chaney, Kostas Daniilidis and Kaushik Roy.

-A collaboration of the C-BRIC teams from Purdue University and University of Pennsylvania.

Acknowledgments

Parts of this code were derived from daniilidis-group/EV-FlowNet and ClementPinard/FlowNetPytorch.