(IEEE TIP) TOPIC: A Parallel Association Paradigm for Multi-Object Tracking under Complex Motions and Diverse Scenes

TOPICTrack proposes a parallel association paradigm, can take full advantage of motion and appearance features.

TOPIC: A Parallel Association Paradigm for Multi-Object Tracking under Complex Motions and Diverse Scenes

Xiaoyan Cao, Yiyao Zheng, Yao Yao, Huapeng Qin, Xiaoyu Cao and Shihui Guo

IEEE TIP

News ✨

• (2025.01) Our paper is online! Please see IEEE.
• (2024.12) TOPICTrack is accepted by IEEE TIP! 🎊🎊🎊
• (2024.10) We expanded the dataset by an order of magnitude, renamed it to BEE24, and made it available at: https://drive.google.com/file/d/1KvqQTZWWhyDsQuEIqQY3XLM_8QlYjDqi/view.
• (2023.08) We provide a complete reproduction tutorial and release the proposed BEE23 dataset as a new benchmark.

Abstract

Video data and algorithms have been driving advances in multi-object tracking (MOT). While existing MOT datasets focus on occlusion and appearance similarity, complex motion patterns are widespread yet overlooked. To address this issue, we introduce a new dataset called BEE24 to highlight complex motions. Identity association algorithms have long been the focus of MOT research. Existing trackers can be categorized into two association paradigms: single-feature paradigm (based on either motion or appearance feature) and serial paradigm (one feature serves as secondary while the other is primary). However, these paradigms are incapable of fully utilizing different features. In this paper, we propose a parallel paradigm and present the Two rOund Parallel matchIng meChanism (TOPIC) to implement it. The TOPIC leverages both motion and appearance features and can adaptively select the preferable one as the assignment metric based on motion level. Moreover, we provide an Attention-based Appearance Reconstruction Module (AARM) to reconstruct appearance feature embeddings, thus enhancing the representation of appearance features. Comprehensive experiments show that our approach achieves state-of-the-art performance on four public datasets and BEE24. Moreover, BEE24 challenges existing trackers to track multiple similar-appearing small objects with complex motions over long periods, which is critical in real-world applications such as beekeeping and drone swarm surveillance. Notably, our proposed parallel paradigm surpasses the performance of existing association paradigms by a large margin, e.g., reducing false negatives by 6% to 81% compared to the single-feature association paradigm. The introduced dataset and association paradigm in this work offer a fresh perspective for advancing the MOT field. The source code and dataset are available at https://github.com/holmescao/TOPICTrack.

Tracking performance

Results on five benchmarks test set

Dataset	HOTA	MOTA	IDF1	AssA	AssR	FP	FN	IDs	Frag
MOT17	63.9	78.8	78.7	64.3	69.9	17,000	101,100	1,515	2,613
MOT20	62.6	72.4	77.6	65.4	70.3	11,000	131,100	869	1,574
DanceTrack	58.3	90.9	58.4	42.3	48.1	5,555	19,246	1,471	2,106
GMOT-40	84.2	95.8	91.5	81.9	84.7	205	227	524	92
BEE23	71.9	86.7	86.3	71.3	77.0	642	634	348	220

Visualization results on five benchmarks test set

Installation

• Install Python dependencies. We utilize Python 3.8 and PyTorch 1.8.0.

  git clone https://github.com/holmescao/TOPICTrack
  cd TOPICTrack
  conda create -n TOPICTrack python=3.8
  conda activate TOPICTrack
  pip install --upgrade pip
  pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html
  cd external/YOLOX/
  pip install -r requirements.txt && python setup.py develop
  cd ../deep-person-reid/
  pip install -r requirements.txt && python setup.py develop
  cd ../fast_reid/
  pip install -r docs/requirements.txt
  

Data preparation

For detector and tracking

Download MOT17, MOT20, DanceTrack, GMOT-40, BEE23 and put them under <TOPICTrack_HOME>/data in the following structure:

data
|-- mot
|   |-- test
|   |-- train
|-- MOT20
|   |-- test
|   |-- train
|-- dancetrack
|   |-- test
|   |-- train
|   |-- val
|-- gmot
|   |-- test
|   |-- train
|-- BEE23
    |-- test
    |-- train

Note that the GMOT-40 contains 4 categories with 10 sequences each. Since the official description does not split the training and test sets for these sequences, this work treats 3 of them (sequentially numbered 0,1,3) for each category as the training set and the remaining 1 (sequentially numbered 2) as the test set.

Then, you need to turn the datasets to COCO format:

cd <TOPICTrack_HOME>
python3 tools/convert_mot17_to_coco.py
python3 tools/convert_mot20_to_coco.py
python3 tools/convert_dance_to_coco.py
python3 tools/convert_gmot_to_coco.py
python3 tools/convert_bee_to_coco.py

For Re-ID model

Download GMOT-40-reid, BEE23-reid and put them under <TOPICTrack_HOME>/fast-reid/datasets in the following structure:

datasets
|-- BEE23
|   |-- bounding_box_test
|   |-- bounding_box_train
|   |-- query
|-- gmot
    |-- bounding_box_test
    |-- bounding_box_train
    |-- query

Here we use Deep-OC-SORT trained Re-ID models on MOT17, MOT20, and DanceTrack.

Model zoo

We provide some pretrained YOLO-X weights and FastReID weights for TOPICTrack.

Please download the required pre-trained weights by yourself and put them into external/weights.

Dataset	HOTA	MOTA	IDF1	FP	FN	Model (Detection)	Model (Re-ID)
MOT17-half-val	69.6	79.8	81.2	3,028	7,612	topictrack_ablation.pth.tar [google]	mot17_sbs_S50.pth [google]
MOT17-test	63.9	78.8	78.7	17,000	101,100	topictrack_mot17.pth.tar [google]	mot17_sbs_S50.pth [google]
MOT20-half-val	57.5	73.0	73.6	28,583	135,945	topictrack_mot17.pth.tar [google]	mot20_sbs_S50.pth [google]
MOT20-test	62.6	72.4	77.6	11,000	131,100	topictrack_mot20.pth.tar [google]	mot20_sbs_S50.pth [google]
DanceTrack-val	55.7	89.3	54.2	12,636	10,567	topictrack_dance.pth.tar [google]	dance_sbs_S50.pth [google]
DanceTrack-test	58.3	90.9	56.6	5,555	19,246	topictrack_dance.pth.tar [google]	dance_sbs_S50.pth [google]
GMOT40-test	84.2	95.8	91.5	205	227	topictrack_gmot.pth.tar [google]	gmot_AGW.pth [google]
BEE23-test	71.9	86.7	86.3	642	634	topictrack_bee.pth.tar [google]	bee_AGW.pth [google]

• For more YOLO-X weights, please refer to the model zoo of ByteTrack.

Training detector

You can use TOPICTrack without training by adopting existing detectors. But we borrow the training guidelines from ByteTrack in case you want work on your own detector.

Download the COCO-pretrained YOLOX weight here and put it under external/weights.

Train ablation model MOT17 half train

You can run the follow command:

sh run/mot17_half_train.sh

Or

python3 tools/train.py -f exps/example/mot/yolox_x_ablation.py -d 1 -b 4 --fp16 -o -c external/weights/yolox_x.pth

Train MOT17 test model

You can run the follow command:

sh run/mot17_train.sh

Or

python3 tools/train.py -f exps/example/mot/yolox_x_mot17_train.py -d 1 -b 4 --fp16 -o -c external/weights/yolox_x.pth

Train MOT20 test model

You can run the follow command:

sh run/mot20_train.sh

Or

python3 tools/train.py -f exps/example/mot/yolox_x_mix_mot20_ch.py -d 1 -b 4 --fp16 -o -c external/weights/yolox_x.pth

Train DanceTrack test model

You can run the follow command:

sh run/dancetrack_train.sh

Or

python3 tools/train.py -f exps/example/mot/yolox_x_dance_train.py -d 1 -b 4 --fp16 -o -c external/weights/yolox_x.pth

Train GMOT-40 test model

You can run the follow command:

sh run/gmot_train.sh

Or

python3 tools/train.py -f exps/example/mot/yolox_x_gmot_train.py -d 1 -b 4 --fp16 -o -c external/weights/yolox_x.pth

Train BEE23 test model

You can run the follow command:

sh run/bee_train.sh

Or

python3 tools/train.py -f exps/example/mot/yolox_x_bee23_train.py -d 1 -b 4 --fp16 -o -c external/weights/yolox_x.pth

Train custom dataset

First, you need to prepare your dataset in COCO format. You can refer to MOT-to-COCO or CrowdHuman-to-COCO. Then, you need to create a Exp file for your dataset. You can refer to the CrowdHuman training Exp file. Don't forget to modify get_data_loader() and get_eval_loader in your Exp file. Finally, you can train bytetrack on your dataset by running:

python3 tools/train.py -f exps/example/mot/your_exp_file.py -d 8 -b 48 --fp16 -o -c pretrained/yolox_x.pth

Training Re-ID model

Here we use Deep-OC-SORT trained Re-ID models on MOT17, MOT20, and DanceTrack, download the weights and put them in external/weights.

The training process for GMOT-40 and BEE23 is as follows.

Train GMOT-40 test model

You can run the follow command:

cd ./fast-reid
sh gmot_train_AGW.sh

Or

cd ./fast-reid
python3 tools/train_net.py --config-file ./configs/gmot/AGW_S50.yml MODEL.DEVICE "cuda:0"

The train results are saved in: fast-reid/logs/gmot_AGW/model_best.pth.

Please rename the model model_best.pth to gmot_AGW.pth, and move it to external/weights/.

Train BEE23 test model

You can run the follow command:

cd ./fast-reid
sh bee_train_AGW.sh

Or

cd ./fast-reid
python3 tools/train_net.py --config-file ./configs/bee/AGW_S50.yml MODEL.DEVICE "cuda:0"

The train results are saved in: fast-reid/logs/bee_AGW/model_best.pth.

Please rename the model model_best.pth to bee23_AGW.pth, and move it to external/weights/.

Evaluation

Please download the pre-trained model mentioned in the Model zoo chapter and put it in external/weights before conduct the following steps.

MOT17

Evaluation on MOT17 half val

You can run the follow command:

sh run/mot17_val.sh

The results are saved in: results/trackers/MOT17-val/mot17_val_post.

Evaluate tracking results:

exp=mot17_val dataset=MOT17 sh eval_metrics.sh

Test on MOT17

You can run the follow command:

sh run/mot17_test.sh

The results are saved in: results/trackers/MOT17-val/mot17_test_post.

MOT20

Evaluation on MOT20 half val

You can run the follow command:

sh run/mot20_val.sh

The results are saved in: results/trackers/MOT20-val/mot20_val_post.

Evaluate tracking results:

exp=mot20_val dataset=MOT20 sh eval_metrics.sh

Test on MOT20

You can run the follow command:

sh run/mot20_test.sh

The results are saved in: results/trackers/MOT20-val/mot20_test_post.

DanceTrack

Evaluation on DanceTrack half val

You can run the follow command:

sh run/dancetrack_val.sh

The results are saved in: results/trackers/DANCE-val/dance_val_post.

Evaluate tracking results:

exp=dance_val dataset=DANCE sh eval_metrics.sh

Test on DanceTrack

You can run the follow command:

sh run/dancetrack_test.sh

The results are saved in: results/trackers/DANCE-val/dance_test_post.

GMOT-40

Evaluation/Test on GMOT-40 half val

GMOT-40 has only training set and test set without dividing the validation set. Therefore the validation set and test set have the same evaluation script. You can run the follow command:

sh run/gmot_test.sh

The results are saved in: results/trackers/GMOT-val/gmot_test_post.

Evaluate tracking results:

exp=gmot_test dataset=GMOT sh eval_metrics.sh

BEE23

Evaluation/Test on BEE23 half val

BEE23 has only training set and test set without dividing the validation set. Therefore the validation set and test set have the same evaluation script. You can run the follow command:

sh run/bee_test.sh

The results are saved in: results/trackers/BEE23-val/bee_test_post.

Evaluate tracking results:

exp=bee_test dataset=BEE23 sh eval_metrics.sh

Demo

# --dataset options: mot17 | mot20 | dance | gmot | BEE23
python3 demo.py --exp_name bee_test --dataset BEE23 --test_dataset

The generated video will be saved in videos/.

Citation

If you find this project useful, please consider to cite our paper. Thank you!

@ARTICLE{10851814,
  author={Cao, Xiaoyan and Zheng, Yiyao and Yao, Yao and Qin, Huapeng and Cao, Xiaoyu and Guo, Shihui},
  journal={IEEE Transactions on Image Processing}, 
  title={TOPIC: A Parallel Association Paradigm for Multi-Object Tracking under Complex Motions and Diverse Scenes}, 
  year={2025},
  volume={},
  number={},
  pages={1-1},
  keywords={Tracking;Measurement;Feature extraction;Detectors;Optimization;Training;Surveillance;Image reconstruction;Focusing;Drones;Multi-object tracking;complex motion patterns;appearance and motion features;parallel association paradigm;appearance reconstruction},
  doi={10.1109/TIP.2025.3526066}}

Acknowledgement

A large part of the code is borrowed from YOLOX, OC-SORT, Deep-OC-SORT and FastReID. Many thanks for their wonderful works.