Skip to content

verlab/SemanticFastForward_ICIP_2016

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Version License

Project

This project is based on the paper Fast-Forward Video Based on Semantic Extraction on the IEEE International Conference on Image Processing (ICIP 2016). It implements a semantic fast-forward method for First-Person videos.

For more information and visual results, please access the project page.

Contact

Authors

Institution

Federal University of Minas Gerais (UFMG)
Computer Science Department
Belo Horizonte - Minas Gerais -Brazil

Laboratory

VeRLab

VeRLab: Laboratory of Computer Vison and Robotics
http://www.verlab.dcc.ufmg.br

Code

Dependencies

  • MATALB 2015a or higher

Usage

  • The project processing is decribed by the following flowchart:

Flowchart

  1. Optical Flow Estimator:

    The first step processing is to estimate the Optical Flow of the Input Video.

    1. First, you should download the Poleg et al. 2014 Flow Estimator code from the link.
    2. Navigate to the download folder and unzip the code.
    3. Into the Vid2OpticalFlowCSV\Example folder, run the command:
Vid2OpticalFlowCSV.exe -v < video_filename > -c < config.xml > -o < output_filename.csv >
Options Description Type Example
< Video_filename > Path and filename of the video. String ~/Data/MyVideos/myVideo.mp4
< config.xml > Path to the configuration XML file. String ../default-config.xml
< output_filename.csv > Path to save the output CSV file. String myVideo.csv
Save the file using the same same name of the video file with extension ".csv".
  1. Semantic Extractor:

    The second step is to extract the semantic information over all frames of the Input video and save it in a CSV file. On the MATLAB console, go to the project folder and run the command:

>> ExtractAndSave(< Video_filename >)
Parameters Description Type Example
< Video_filename > Path and filename of the video. String ~/Data/MyVideos/Video.mp4
  1. Calculate Speed-up rates

    To calculate the speed-up rates for each type of segment, on the MATLAB console, go to the project folder and run the following commands:

>> [~, < Num_non_semantic_Frames >, < Num_semantic_frames >] = GetSemanticRanges(< Semantic_Data_MAT_filename >);
Parameters Description Type Example
< Semantic_Data_MAT_filename > File created by the step 2 String '../example_face_extracted.mat'
Output Description Type Example
< Num_non_semantic_Frames > Number of frames in the Non-Semantic segments. (Will be used below.) Integer Tns
< Num_semantic_Frames > Number of frames in the Semantic segments. (Will be used below.) Integer Ts
  1. Calculate Speed-up rates

    To calculate the speed-up rates for each type of segment, on the MATLAB console, go to the project folder and run the command:

>> SpeedupOptimization( < Num_non_semantic_Frames >, < Num_semantic_frames >, < Desired_speedup >, < Max_speedup>, < lambda_1 >, < lambda_2 >, < show_plot > )
Parameters Description Type Example
< Num_non_semantic_Frames > Number of frames in the Non-Semantic segments. Integer 7643
< Num_semantic_Frames > Number of frames in the Semantic segments. Integer 12935
< Desired_speedup > Desired speed-up rate to the whole video. Integer 10
< Max_speedup > Maximum allowed jump. Integer 10
< Lambda_1 > Value of Lambda 1 in the optimization function. Integer 40
< Lambda_2 > Value of Lambda 1 in the optimization function. Integer 8
< show_plot > Flag to show the search space create by the optimization function. Boolean false
  1. Create Experiment:

    To run the code, you should create an experiment entry. On a text editor, add a case to the GetVideoDetails function in the file SemanticSequenceLibrary.m:

function [videoFile, startInd, endInd, filename, fps] = GetVideoDetails(video_dir,exp_name)

 ...

 case < Experiment_name >
     filename = < video_filename >;
     startInd = < start_index_frame > ;
     endInd   = < final_index_frame >;
     fps      = < video_frames_per_second >;
					
  ... 
  
Fields Description Type Example
< Experiment_name > Name to identify the experiment. String MyVideo
< video_filename > Filename to the video. String myVideo.mp4
< start_index_frame > Frame index to start the processing. Integer 1
< final_index_frame > Frame intex to stop the processing. Integer 16987
< video_frames_per_second > Frames per second of the video. Integer 30
  1. Semantic Fast-Forward:

    After the previous steps, you are ready to accelerate the Input Video. On MATLAB console, go to the project folder and run the command:

>> SpeedupVideo(< Video_dir >, < Experiment_name >, < Semantic_extractor_name >, < Semantic_speedup >, < Non_semantic_speedup >, < Shakiness_weights >, < Velocity_weights >, < Appearance_weights >, < Semantic_weights > )
Parameters Description Type Example
< Video_dir > Path to the folder where the video file is. String ~/Data/MyVideos
< Experiment_name > Name set in the SemanticSequenceLibrary.m file. String My_Video
< Semantic_extractor_name > Semantic extractor algorithm used in the Semantic Extractor step. String 'face' or 'pedestrian'
< Semantic_speedup > Desired speed-up assign to the semantic segments of the input video. Integer 3
< Non_semantic_speedup > Desired speed-up assign to the semantic segments of the input video. Integer 14
< Shakiness_weights > Tuple of weights related to the shakiness term in the edge weight formulation. [Integer, Integer] [10,250]
< Velocity_weights > Tuple of weights related to the shakiness term in the edge weight formulation. [Integer, Integer] [50,3000]
< Appearance_weights > Tuple of weights related to the shakiness term in the edge weight formulation. [Integer, Integer] [0,100]
< Semantic_weights > Tuple of weights related to the shakiness term in the edge weight formulation. [Integer, Integer] [500,8]

Acknowledgements

  1. EgoSampling: Y. Poleg, T. Halperin, C. Arora, S. Peleg, Egosampling: Fast-forward and stereo for egocentric videos, in: IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 4768-4776. doi:10.1109/CVPR.2015.7299109.
  2. LK_Blocked_Optical_Flow: Y. Poleg, C. Arora, S. Peleg, Temporal segmentation of egocentric videos, in: IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2537-2544. doi:10.1109/CVPR.2014.325.
  3. NPD_Face_Detector: S. Liao, A. K. Jain, S. Z. Li, A fast and accurate unconstrained face detector, IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (2) (2016), pp. 211-223. doi:10.1109/TPAMI.2015.2448075.

Citation

If you are using it for academic purposes, please cite:

W. L. S. Ramos, M. M. Silva, , M. F. M. Campos, E. R. Nascimento, Fast-forward video based on semantic extraction, in: -IEEE International Conference on Image Processing, Phoenix, AZ, 2016, pp. 3334-3338. doi:10.1109/ICIP.2016.7532977.

Bibtex entry

@InProceedings{Ramos2016,
author = {W. L. S. Ramos and M. M. Silva and M. F. M. Campos and E. R. Nascimento},
booktitle = {IEEE International Conference on Image Processing},
title = {Fast-forward video based on semantic extraction},
year = {2016},
month = {Sept},
Address = {Phoenix, AZ},
pages = {3334-3338},
doi = {10.1109/ICIP.2016.7532977}
}

Enjoy it.