DBG-dense-boundary-generator

By Chuming Lin*, Jian Li*, Yabiao Wang, Ying Tai, Donghao Luo, Zhipeng Cui, Chengjie Wang, Jilin Li, Feiyue Huang, Rongrong Ji.

*indicates equal contributions

Update

• 2019.11.12: Release tensorflow-version DBG inference code.
• 2019.11.11: DBG is accepted by AAAI2020.
• 2019.11.08: Our ensemble DBG ranks No.1 on ActivityNet

Introduction

In this repo, we propose a novel and unified action detection framework, named DBG, with superior performance over the state-of-the-art action detectors BSN and BMN. You can use the code to evaluate our DBG for action proposal generation or action detection. For more details, please refer to our paper Fast Learning of Temporal Action Proposal via Dense Boundary Generator!

Contents

• Paper Introduction
• Prerequisites
• Getting Started
• Citation
• Contact

Paper Introduction

This paper introduces a novel and unified temporal action proposal generator named Dense Boundary Generator (DBG). In this work, we propose dual stream BaseNet to generate two different level and more discriminative features. We then adopt a temporal boundary classification module to predict precise temporal boundaries, and an action-aware completeness regression module to provide reliable action completeness confidence.

ActivityNet1.3 Results

THUMOS14 Results

Qualitative Results

Prerequisites

• Tensorflow == 1.9.0
• Python == 3.6
• NVIDIA GPU == Tesla P40
• Linux CUDA 9.0 CuDNN

Getting Started

Installation

Clone the github repository. We will call the cloned directory as $DBG_ROOT.

git clone https://github.com/TencentYoutuResearch/ActionDetection-DBG.git
cd ActionDetection-DBG
export CUDA_VISIBLE_DEVICES=0

Download Datasets

Prepare ActivityNet 1.3 dataset. You can use official ActivityNet downloader to download videos from the YouTube. Some videos have been deleted from YouTube，and you can also ask for the whole dataset by email.

Extract visual feature, we adopt TSN model pretrained on the training set of ActivityNet, Please refer this repo TSN-yjxiong to extract frames and optical flow and refer this repo anet2016-cuhk to find pretrained TSN model.

For convenience of training and testing, we rescale the feature length of all videos to same length 100, and we provide the 19993 rescaled feature at here Google Cloud or 微云.

Compile Proposal Feature Generation Layer

Please make sure your Tensorflow==1.9.0 and Python==3.6

cd custom_op/src
make

Runing of DBG

Pretrained model is included in output/pretrained_model and set parameters on config.yaml. Please check the feat_dir in config.yaml and use a script to run DBG.

bash auto_run.sh

This script contains:

1. Training

python train.py ./config.yaml

2. Testing

python test.py ./config.yaml

3. Evaluating

python post_processing.py output/result output/result_proposal.json
python eval.py output/result_proposal.json

Citation

If you find DBG useful in your research, please consider citing:

@inproceedings{DBG2020arXiv,
  author    = {Chuming Lin*, Jian Li*, Yabiao Wang, Ying Tai, Donghao Luo, Zhipeng Cui, Chengjie Wang, Jilin Li, Feiyue Huang, Rongrong Ji},
  title     = {Fast Learning of Temporal Action Proposal via Dense Boundary Generator},
  booktitle   = {AAAI Conference on Artificial Intelligence},
  year      = {2020},
}

Contact

For any question, please file an issue or contact

Jian Li: swordli@tencent.com