Awesome-Surgical-Video-Understanding

There are compilations of surgery-related tasks, datasets, and papers.

Contents

• Dataset
  • Endoscopic Surgery
    • Public Dataset
    • Private Dataset
  • Egocentric Open Surgery
  • Ophthalmic Surgery
• Paper
  • Surgical Phase Recognition
  • Surgical Triplet Recognition
  • Surgical Tool Detection
  • Segmentation
  • Surgical Scene Reconstruction
  • Surgical Video Generation
  • Surgical VQA
  • Surgical Type Recognition
  • Surgical Pretraining
• Comprehensive Journals

Dataset

Endoscopic Surgery

Public Dataset

Dataset	Paper	Link	Procedure	Task	Type
Cholec80	Paper	Link	Cholecystectomy Surgery	Surgical Phase Recognition	Video-level
Cholec80-CSV	Paper	Link	Cholecystectomy Surgery	Critical View of Safety (CVS)	Video-level
M2CAI16-Workflow	Paper	Link	Cholecystectomy Surgery	Surgical Phase Recognition	Video-level
M2CAI16-Tool	Paper	Link	Cholecystectomy Surgery	Surgical Tool Recognition	Video-level
CholecT50	Paper	Link	Cholecystectomy Surgery	Surgical Triplet Recognition	Video-level
HeiChole	Paper	Link	Cholecystectomy Surgery	Surgical Phase/Action/Instrument Recognition / Skill Assessment	Video-level
Pit-Vis	Paper	Link	Pituitary Surgery	Surgical Step/Instrument Recognition	Video-level
PSI-AVA	Paper	Link	Prostatectomy Surgery	Surgical Phase/Step/Action/Instrument Recognition	Video-level
GraSP	Paper	Link	Prostatectomy Surgery	Surgical Phase/Step/Action/Instrument Recognition / Surgical Instrument Segmentation	Video-level
AutoLaparo	Paper	Link	Hysterectomy Surgery	Surgical Phase Recognition / Motion Prediction	Video-level
MultiBypass140	Paper	Link	Laparoscopic Roux-en-Y Gastric Bypass Surgery	Surgical Phase/Step Recognition	Video-level
SurgToolLoc22	Paper	Link	DaVinci Robotic Surgery	Surgical Tool Recognition/Detection	Video-level
SurgT	Paper	Link	-	Surgical Tissue Tracking	Video-level (30s)
SSG-VQA	Paper	Link	Cholecystectomy Surgery	Visual Question Answering	Image-level
EndoVis-18-VQA	Paper	Link	Robotic Nephrectomy Surgery	Visual Question Answering	Image-level
Cholec80-VQA	Paper	Link	Cholecystectomy Surgery	Visual Question Answering	Image-level
Endoscapes	Paper	Link	Cholecystectomy Surgery	Segmentation/Detection/CVS	Image-level
StereoMIS	Paper	Link	DaVinci Robotic Surgery	Surgical Scene Reconstruction
EndoNeRF	Paper	Link	DaVinci robotic prostatectomy	Surgical Scene Reconstruction

Private Dataset

Dataset	Paper	Procedure	Task	Type
Laparo425	Paper	Laparoscopic Surgery	Surgical Type Recognition	Video-level
MultiCholec2022	Paper	Cholecystectomy Surgery	Surgical Phase Recognition	Video-level
Laparo23.3M	Paper	Laparoscopic Surgery	-	Video-level

Egocentric Open Surgery

Dataset	Paper	Link	Type	Task
EgoSurgery-Tool	Paper	Link	Distinct Surgical Procedures	Surgical Tool/Hand Detection
EgoSurgery-Phase	Paper	Link	Distinct Surgical Procedures	Surgical Phase Recognition

Paper

Surgical Phase Recognition

Paper	Surgery	Image	Link	Code	Type
EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos (TMI-16)	Endoscopic Surgery		Link		Vision
SV-RCNet: Workflow Recognition From Surgical Videos Using Recurrent Convolutional Network (TMI-18)	Endoscopic Surgery		Link	Code	Vision
Hard Frame Detection and Online Mapping for Surgical Phase Recognition (MICCAI-19)	Endoscopic Surgery		Link	Code	Vision
Multi-task recurrent convolutional network with correlation loss for surgical video analysis (MIA-20)	Endoscopic Surgery		Link	Code	Vision
TeCNO: Surgical Phase Recognition with Multi-stage Temporal Convolutional Networks (MICCAI-20)	Endoscopic Surgery		Link	Code	Vision
Temporal Memory Relation Network for Workflow Recognition from Surgical Video (TMI-21)	Endoscopic Surgery		Link	Code	Vision
Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformer (MICCAI-21)	Endoscopic Surgery		Link	Code	Vision
Federated Cycling (FedCy): Semi-supervised Federated Learning of Surgical Phases (TMI-22)	Endoscopic Surgery		Link		Vision
SKiT: a Fast Key Information Video Transformer for Online Surgical Phase Recognition (ICCV-23)	Endoscopic Surgery		Link	Code	Vision
Surgformer: Surgical Transformer with Hierarchical Temporal Attention for Surgical Phase Recognition (MICCAI-24)	Endoscopic Surgery		Link	Code	Vision
Label-guided Teacher for Surgical Phase Recognition via Knowledge Distillation (MICCAI-24)	Endoscopic Surgery		Link		Vision
EgoSurgery-Phase: A Dataset of Surgical Phase Recognition from Egocentric Open Surgery Videos (MICCAI-24)	Egocentric Open Surgery		Link	Code	Vision
HecVL: Hierarchical Video-Language Pretraining for Zero-shot Surgical Phase Recognition (MICCAI-24)	Endoscopic Surgery		Link		Vision-Language
Jumpstarting Surgical Computer Vision (MICCAI-24)	Endoscopic Surgery		Link		Vision
SR-Mamba: Effective Surgical Phase Recognition with State Space Model	Endoscopic Surgery		Link	Code	Vision
LoViT: Long Video Transformer for surgical phase recognition (MIA-25)	Endoscopic Surgery		Link	Code	Vision
Surgical Video Workflow Analysis via Visual-Language Learning (npj-under review)	Endoscopic Surgery		Link	Code	Vision-Language
Neural Finite-State Machines for Surgical Phase Recognition	Endoscopic Surgery		Link		Vision
SWAG: Long-term Surgical Workflow Prediction with Generative-based Anticipation		Link	Code	Vision
Benchmarking and Enhancing Surgical Phase Recognition Models for Robotic-Assisted Esophagectomy		Link		Vision

Surgical Triplet Recognition

Paper	Surgery	Image	Link	Code	Type
Tail-Enhanced Representation Learning for Surgical Triplet Recognition	Endoscopic Surgery		Link	Code	Vision

Surgical Tool Detection

Paper	Surgery	Image	Link	Code	Type
EgoSurgery-Tool: A Dataset of Surgical Tool and Hand Detection from Egocentric Open Surgery Videos	Endoscopic Surgery		Link	Code	Vision

Segmentation

Paper	Surgery	Image	Link	Code	Type
Image Compositing for Segmentation of Surgical Tools without Manual Annotations (TMI-21)	Endoscopic Surgery		Link	Code	Vision

Surgical Scene Reconstruction

Paper	Surgery	Image	Link	Code	Type
Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery (MICCAI-22)	Endoscopic Surgery		Link	Code	Vision
EndoSurf: Neural Surface Reconstruction of Deformable Tissues with Stereo Endoscope Videos (MICCAI-23 Oral)	Endoscopic Surgery		Link	Code	Vision
Deform3DGS: Flexible Deformation for Fast Surgical Scene Reconstruction with Gaussian Splatting (MICCAI-24)	Endoscopic Surgery		Link	Code	Vision
Free-SurGS: SfM-Free 3D Gaussian Splatting for Surgical Scene Reconstruction (MICCAI-24)	Endoscopic Surgery		Link	Code	Vision
LGS: A Light-weight 4D Gaussian Splatting for Efficient Surgical Scene Reconstruction (MICCAI-24)	Endoscopic Surgery		Link	Code	Vision
SurgicalGaussian: Deformable 3D Gaussians for High-Fidelity Surgical Scene Reconstruction (MICCAI-24)	Endoscopic Surgery		Link	Code	Vision

Surgical Video Generation

Paper	Surgery	Image	Link	Code	Type
See, Predict, Plan: Diffusion for Procedure Planning in Robotic Surgical Videos (MICCAI-24)	Endoscopic Surgery		Link		Vision

Surgical VQA

Paper	Surgery	Image	Link	Code	Type
Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformer (MICCAI-22)	Endoscopic Surgery		Link	Code	Vision-Language
Advancing Surgical VQA with Scene Graph Knowledge (IPCAI-24)	Endoscopic Surgery		Link	Code	Vision-Language

Surgical Type Recognition

Paper	Surgery	Image	Link	Code	Type
Future-State Predicting LSTM for Early Surgery Type Recognition (TMI-20)	Endoscopic Surgery		Link		Vision

Surgical Pretraining

Paper	Surgery	Image	Link	Code	Type
LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning	Endoscopic Surgery		Link		Vision-Language
Surgical-LLaVA: Toward Surgical Scenario Understanding via Large Language and Vision Models	Endoscopic Surgery		Link		Vision-Language
OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretraining	Ophthalmic Surgery		Link	Code	Vision-Language

Comprehensive Journals

Paper	Journal	Image	Link	Code
Surgical gestures as a method to quantify surgical performance and predict patient outcomes (Dani Kiyasseh et al.)	npj Digital Medicine 2022		Link	Code
A vision transformer for decoding surgeon activity from surgical videos (Dani Kiyasseh et al.)	Nature Biomedical Engineering 2023		Link	Code
Human visual explanations mitigate bias in AI-based assessment of surgeon skills (Dani Kiyasseh et al.)	npj Digital Medicine 2023		Link	Code