You can easily use colaboratory to have a try.
Our model can run 30+FPS for 2X 720p interpolation on a 2080Ti GPU. We also support CPU device including MacOS system. Currently our method supports multi-frame interpolation between a pair of images and 2X interpolation for video. Everyone is welcome to use this alpha version and make suggestions!
16X interpolation results from two input images:
We propose a real-time intermediate flow estimation algorithm (RIFE) for video frame interpolation (VFI). Most existing methods first estimate the bi-directional optical flows, and then linearly combine them to approximate intermediate flows, leading to artifacts around motion boundaries. We design an intermediate flow model named IFNet that can directly estimate the intermediate flows from coarse to fine. We then warp the input frames according to the estimated intermediate flows and employ a fusion process to compute final results. Based on our proposed leakage distillation, RIFE can be trained end-to-end and achieve excellent performance. Experiments demonstrate that RIFE is significantly faster than existing flow-based VFI methods and achieves state-of-the-art index on several benchmarks.
$ pip3 install tqdm
$ pip3 install torch
$ pip3 install numpy
$ pip3 install opencv-python
- Download the pretrained models from here
(We also provide 百度网盘 source. 链接: https://pan.baidu.com/s/17tHd-syovvRGP2C6UVPsIw 提取码: 5ha7)
- Unzip and move the pretrained parameters to train_log/*.pkl
The models under different setting is coming soon. Image Interpolation
$ python3 inference_img.py --img img0.png img1.png --times=4
(2^4=16X interpolation results)
$ ffmpeg -r 10 -f image2 -i output/img%d.png -s 448x256 -c:v libx264 -pix_fmt yuv420p output/slomo.mp4 -q:v 0 -q:a 0
(generate a slomo mp4 video based on two input images)
$ ffmpeg -r 10 -f image2 -i output/img%d.png -s 448x256 -vf "split[s0][s1];[s0]palettegen=stats_mode=single[p];[s1][p]paletteuse=new=1" output/slomo.gif
Video 2x Interpolation
$ python3 inference_mp4_2x.py --video video.mp4
(generate video_2x.mp4, you can use this script recursively to get 4X, 8X, 16X...)
$ python3 inference_mp4_2x.py --video myvideo.mp4 --montage
(if you want to montage the origin video)
We will release our training and benchmark validation code soon.
Download Vimeo90K dataset at ./vimeo_interp_test
$ python3 Vimeo90K_benchmark.py
(You will get 35.695PSNR and 0.9788SSIM)
@article{huang2020rife,
title={RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation},
author={Huang, Zhewei and Zhang, Tianyuan and Heng, Wen and Shi, Boxin and Zhou, Shuchang},
journal={arXiv preprint arXiv:2011.06294},
year={2020}
}
Optical Flow: ARFlow pytorch-liteflownet RAFT
Video Interpolation: DAIN CAIN AdaCoF-pytorch