Can Shuffling Video Benefit Temporal Bias Problem for Temporal Grounding

Code for ECCV 2022 paper "Can Shuffling Video Benefit Temporal Bias Problem: A Novel Training Framework for Temporal Grounding" [arxiv].

Installation

We provide the environment file for anaconda.

You can build the conda environment simply by,

conda env create -f environment.yml

Dataset Preparation

Features and Pretrained Models

You can download our features for Charades-STA and ActivityNet Captions and the pretrained models of our method on re-divided splits by links (box drive, google drive, baidu drive code:yfar ).

(For ActivityNet Captions, we extract the i3d features from the original videos using an open implementation of I3D, with stride 16 and fps 16.)

Please put the video feature files 'VID.npy' into the directories data/Charades/i3d_feature and data/ANet/i3d_feature, respectively.

Please put the pretrained models into the directories grounding/ckp/charades_cd and grounding/ckp/anet_cd, respectively.

Word Embeddings

For Charades-STA, we directly provide the word embeddings files in this github repositories. You don't need to do anything else.

For ActivityNet Captions, due to the limitation of the file size of github, you need to download the word embeddings from the (box drive, google drive, baidu drive code:yfar ), and put the word embeddings into the directory data/ANet/words.

Quick Start

conda activate HLTI
cd grounding

Charades-CD

Train:

python train.py --gpu_id=0 --cfg charades_cd_i3d.yml --alias one_name

The checkpoints and prediction results will be saved in grounding/runs/DATASET/

Evaluate:

python test.py --gpu_id=0 --cfg charades_cd_i3d.yml --alias test

You can change the model to be evaluated in the corresponding config file. By default, test.py will use the pre-trained model provided by us.

ActivityNet-CD

Train:

python train.py --gpu_id=0 --cfg anet_cd_i3d.yml --alias one_name

Evaluate:

python test.py --gpu_id=0 --cfg anet_cd_i3d.yml --alias test

About Pretrained Models

We provide the corresponding prediction results, parameter setting, and evaluation result files in grounding/ckp for both datasets.

Baseline

We also provide the implementation of the baseline (QAVE).

Charades-CD

Train:

python train_baseline.py --gpu_id=0 --cfg charades_cd_i3d.yml --alias one_name

Evaluate:

python test_baseline.py --gpu_id=0 --cfg charades_cd_i3d.yml --alias test

Please determine the model to be evaluated in the corresponding config file.

ActivityNet-CD

Train:

python train_baseline.py --gpu_id=0 --cfg anet_cd_i3d.yml --alias one_name

Evaluate:

python test_baseline.py --gpu_id=0 --cfg anet_cd_i3d.yml --alias test

Citation

Please cite our papers if you find them useful for your research.

@inproceedings{hao2022shufflevideos,
  author    = {Hao, Jiachang and Sun, Haifeng and Ren, Pengfei and Wang, Jingyu and Qi, Qi and Liao, Jianxin},
  title     = {Can Shuffling Video Benefit Temporal Bias Problem: A Novel Training Framework for Temporal Grounding},
  booktitle = {European Conference on Computer Vision},
  year      = {2022},
}

The baseline QAVE is:

@article{hao2022qave,
  title={Query-Aware Video Encoder for Video Moment Retrieval},
  author={Hao, Jiachang and Sun, Haifeng and Ren, Pengfei and Wang, Jingyu and Qi, Qi and Liao, Jianxin},
  journal={Neurocomputing},
  year={2022},
  publisher={Elsevier}
}