A Web App for Generating Events from Upsampled Video Frames & Visualization of Events from Live Video of a Webcam!
Siddhant Sahu | Daniel Gehrig | Nico Messikommer
Table of Contents
The goal of this project is to make ESIM and VID2E available to researchers who do not possess a real event camera. For this we want to deploy VID2E and ESIM as an interactive web app. The app should be easy to use and have the following functional requirements:
- Generation of events from video through dragging and dropping a video into the browser. The resulting events should be downloadable as raw events and rendered video.
- Incorporate video interpolation via an existing video interpolation method (e.g. Super SloMo) before event generation with ESIM.
- Visualization and inspection of the event stream in the browser.
- The event generation should be configurable by changing ESIM parameters.
Follow these steps in order to make this app run in your local system.
This code base assumes you have access to a NVIDIA GPU in your system with proper drivers installed along with the cuda-toolkit
version 10.1
.
Make sure that you have followed the Instalation with Anaconda Instruction and have created the vid2e
Conda environment with the esim_torch
package installed in this environment.
- Activate the Conda environment
conda activate vid2e
conda list | grep 'esim'
The above command should output the following
esim-cuda 0.0.0 pypi_0 pypi
esim-py 0.0.1 pypi_0 pypi
- Install the
streamlit
package in this Conda environment, along with a couple of additional packages.
pip install streamlit stqdm numba h5py
If you want to run this app locally on your system then follow these steps.
- Run the App
streamlit run web_app.py
- Then you can access the app in Local URL: http://localhost:8501
When using the 'video upload' feature, the uploaded file is saved in the 'data/original/video_upload' directory. Subsequently, upon clicking the 'Generate Events' button, two new directory, namely, 'data/upsampled/video_upload' and 'data/events/video_upload' are created. Additionally, after the process has finished, you will see a 'Download Events' button, you can click this button to download the events. Events can be downloaded in three different formats: as .h5 file, as .npz or as a rendered video.
To run this web app on Google Colab follow these steps in order.
Make sure you select the Run time type
as GPU
in Google Colab
- Register and create an account in the remote.it.
- Install remote.it service in your Google Colab instance.
!curl -LkO https://raw.githubusercontent.com/remoteit/installer/master/scripts/auto-install.sh
! chmod +x ./auto-install.sh
! sudo ./auto-install.sh
- Install Miniconda in your Google Colab Instance
! wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh
! chmod +x Miniconda3-py37_4.8.2-Linux-x86_64.sh
! bash ./Miniconda3-py37_4.8.2-Linux-x86_64.sh -b -f -p /usr/local
- Update Python Path
import sys
sys.path.insert(0,'/usr/local/lib/python3.7/site-packages/')
- Install Streamlit package
!pip install streamlit
- Run the remote.it service
!sudo connectd_installer
- In the menu selection choose
1
- Enter your remote.it
email address
andpassword
- Enter the name of your device
esim
- Choose
1
Attach/reinstall a remote.it Service to an application
- Protocol Selection Menu, Choose
2
,Web (HTTP) on port 80
- Enter a name for this remote.it service,
esim
- Main menu, choose,
5
Exit
- Mount your Google Drive to this Google Colab Instance
- Change directory to where to want this project to reside.
- Clone the repo
git clone https://github.com/uzh-rpg/rpg_vid2e.git
- Check if the
esim_torch
package is installed
import esim
print(esim_torch.__path__)
- Built the
esim_torch
package withpybind11
pip install esim_torch/
- Change directory inside
web_app
and run the app on port number80
as a webservice which can then be accessed for the remote.it service.
!streamlit run --server.port 80 web_app.py&>/dev/null&
Finaly, you can navigate your remote.it dashboard and open these service and use the app!
Note: Remeber when running this app on Google Colab you need to change the paths for input and output directories for the upsampling to the specific directory in Google Drive and also other paths and esim import.
- Add video upload dragging and dropping a video into the browser.
- Add video interpolation via an existing video interpolation method (e.g. Super SloMo) before event generation with ESIM.
- Add events download button (output format)
- HDF5
- NPZ
- Rendered Video
- Add webcam functionality
- Run this app on Google Colab