English | 简体中文

RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptions

Lingdong Kong¹ Shaoyuan Xie² Hanjiang Hu³ Benoit Cottereau⁴ Lai Xing Ng⁵ Wei Tsang Ooi¹
¹新加坡国立大学 ²华中科技大学 ³卡耐基梅隆大学 ⁴CNRS ⁵A*STAR

项目概览

RoboDepth is a comprehensive evaluation benchmark designed for probing the robustness of monocular depth estimation algorithms. It includes 18 common corruption types, ranging from weather and lighting conditions, sensor failure and movement, and noises during data processing.

版本更新

[2023.01] - The NYUDepth2-C dataset is ready to be downloaded! See here for more details.
[2023.01] - Evaluation server for Track 2 (fully-supervised depth estimation) is available on this page.
[2023.01] - Evaluation server for Track 1 (self-supervised depth estimation) is available on this page.
[2022.11] - We are organizing the 1st RoboDepth Competition at ICRA 2023. Join the challenge today! 🙋
[2022.11] - The KITTI-C dataset is ready to be downloaded! See here for more details.

大纲

安装
数据准备
开始实验
模型库
鲁棒性基线
特性分析
生成"损坏"数据
更新计划
引用
许可
致谢

安装

Kindly refer to 安装.md for the installation details.

数据准备

Our datasets are hosted by OpenDataLab.

OpenDataLab is a pioneering open data platform for the large AI model era, making datasets accessible. By using OpenDataLab, researchers can obtain free formatted datasets in various fields.

RoboDepth Benchmark

Kindly refer to 数据准备.md for the details to prepare the ¹KITTI, ²KITTI-C, ³Cityscapes, ⁴NYUDepth2, and ⁵NYUDepth2-C datasets.

Competition @ ICRA 2023

Kindly refer to this page for the details to prepare the training and evaluation data associated with the 1st RoboDepth Competition at the 40th IEEE Conference on Robotics and Automation (ICRA 2023).

开始实验

Kindly refer to 开始实验.md to learn more usage about this codebase.

模型库

🚘 - 室外深度估计

自监督深度估计

MonoDepth2, ICCV 2019. ^[Code]

DepthHints, ICCV 2019. ^[Code]

MaskOcc, arXiv 2019. ^[Code]

DNet, IROS 2020. ^[Code]

SGDepth, ECCV 2020. ^[Code]

CADepth, 3DV 2021. ^[Code]

TC-Depth, 3DV 2021. ^[Code]

HR-Depth, AAAI 2021. ^[Code]

Insta-DM, AAAI 2021. ^[Code]

DIFFNet, BMVC 2021. ^[Code]

ManyDepth, CVPR 2021. ^[Code]

EPCDepth, ICCV 2021. ^[Code]

FSRE-Depth, ICCV 2021. ^[Code]

R-MSFM, ICCV 2021. ^[Code]

MonoViT, 3DV 2022. ^[Code]

DepthFormer, CVPR 2022. ^[Code]

DynaDepth, ECCV 2022. ^[Code]

DynamicDepth, ECCV 2022. ^[Code]

RA-Depth, ECCV 2022. ^[Code]

Dyna-DM, arXiv 2022. ^[Code]

TriDepth, WACV 2023. ^[Code]

FreqAwareDepth, WACV 2023. ^[Code]

Lite-Mono, CVPR 2023. ^[Code]

全监督深度估计

AdaBins, CVPR 2021. ^[Code]

NeWCRFs, CVPR 2022. ^[Code]

DepthFormer, arXiv 2022. ^[Code]

GLPDepth, arXiv 2022. ^[Code]

半监督深度估计

MaskingDepth, arXiv 2022. ^[Code]

🏠 - 室内深度估计

自监督深度估计

P²Net, ECCV 2020. ^[Code]

EPCDepth, ICCV 2021. ^[Code]

全监督深度估计

BTS, arXiv 2019. ^[Code]

AdaBins, CVPR 2021. ^[Code]

DPT, ICCV 2021. ^[Code]

SimIPU, AAAI 2022. ^[Code]

NeWCRFs, CVPR 2022. ^[Code]

P3Depth, CVPR 2022. ^[Code]

DepthFormer, arXiv 2022. ^[Code]

GLPDepth, arXiv 2022. ^[Code]

BinsFormer, arXiv 2022. ^[Code]

半监督深度估计

MaskingDepth, arXiv 2022. ^[Code]

鲁棒性基线

📊 指标: The following metrics are consistently used in our benchmark:

Absolute Relative Difference (the lower the better): $\text{Abs Rel} = \frac{1}{|D|}\sum_{pred\in D}\frac{|gt - pred|}{gt}$ .
Accuracy (the higher the better): $\delta_t = \frac{1}{|D|}|{\ pred\in D | \max{(\frac{gt}{pred}, \frac{pred}{gt})< 1.25^t}}| \times 100\%$ .
Depth Estimation Error (the lower the better):
- $\text{DEE}_1 = \text{Abs Rel} - \delta_1 + 1$ ;
- $\text{DEE}_2 = \frac{\text{Abs Rel} - \delta_1 + 1}{2}$ ;
- $\text{DEE}_3 = \frac{\text{Abs Rel}}{\delta_1}$ .
The second Depth Estimation Error term ($\text{DEE}_2$) is adopted as the main indicator for evaluating model performance in our RoboDepth benchmark. The following two metrics are adopted to compare between models' robustness:
- mCE (the lower the better): The average corruption error (in percentage) of a candidate model compared to the baseline, which is calculated among all corruption types across five severity levels.
- mRR (the higher the better): The average resilience rate (in percentage) of a candidate model compared to its "clean" performance, which is calculated among all corruption types across five severity levels.

⚙️ 注释: Symbol ^⭐ denotes the baseline model adopted in mCE calculation.

KITTI-C

Model	Modality	mCE (%)	mRR (%)	Clean	Bright	Dark	Fog	Frost	Snow	Contrast	Defocus	Glass	Motion	Zoom	Elastic	Quant	Gaussian	Impulse	Shot	ISO	Pixelate	JPEG
MonoDepth2_R18^⭐	Mono	100.00	84.46	0.119	0.130	0.280	0.155	0.277	0.511	0.187	0.244	0.242	0.216	0.201	0.129	0.193	0.384	0.389	0.340	0.388	0.145	0.196
MonoDepth2_R18+nopt	Mono	119.75	82.50	0.144	0.183	0.343	0.311	0.312	0.399	0.416	0.254	0.232	0.199	0.207	0.148	0.212	0.441	0.452	0.402	0.453	0.153	0.171
MonoDepth2_R18+HR	Mono	106.06	82.44	0.114	0.129	0.376	0.155	0.271	0.582	0.214	0.393	0.257	0.230	0.232	0.123	0.215	0.326	0.352	0.317	0.344	0.138	0.198
MonoDepth2_R50	Mono	113.43	80.59	0.117	0.127	0.294	0.155	0.287	0.492	0.233	0.427	0.392	0.277	0.208	0.130	0.198	0.409	0.403	0.368	0.425	0.155	0.211
MaskOcc	Mono	104.05	82.97	0.117	0.130	0.285	0.154	0.283	0.492	0.200	0.318	0.295	0.228	0.201	0.129	0.184	0.403	0.410	0.364	0.417	0.143	0.177
DNet_R18	Mono	104.71	83.34	0.118	0.128	0.264	0.156	0.317	0.504	0.209	0.348	0.320	0.242	0.215	0.131	0.189	0.362	0.366	0.326	0.357	0.145	0.190
CADepth	Mono	110.11	80.07	0.108	0.121	0.300	0.142	0.324	0.529	0.193	0.356	0.347	0.285	0.208	0.121	0.192	0.423	0.433	0.383	0.448	0.144	0.195
HR-Depth	Mono	103.73	82.93	0.112	0.121	0.289	0.151	0.279	0.481	0.213	0.356	0.300	0.263	0.224	0.124	0.187	0.363	0.373	0.336	0.374	0.135	0.176
DIFFNet_HRNet	Mono	94.96	85.41	0.102	0.111	0.222	0.131	0.199	0.352	0.161	0.513	0.330	0.280	0.197	0.114	0.165	0.292	0.266	0.255	0.270	0.135	0.202
ManyDepth_single	Mono	105.41	83.11	0.123	0.135	0.274	0.169	0.288	0.479	0.227	0.254	0.279	0.211	0.194	0.134	0.189	0.430	0.450	0.387	0.452	0.147	0.182
FSRE-Depth	Mono	99.05	83.86	0.109	0.128	0.261	0.139	0.237	0.393	0.170	0.291	0.273	0.214	0.185	0.119	0.179	0.400	0.414	0.370	0.407	0.147	0.224
MonoViT_MPViT	Mono	79.33	89.15	0.099	0.106	0.243	0.116	0.213	0.275	0.119	0.180	0.204	0.163	0.179	0.118	0.146	0.310	0.293	0.271	0.290	0.162	0.154
MonoViT_MPViT+HR	Mono	70.79	90.67	0.090	0.097	0.221	0.113	0.217	0.253	0.113	0.146	0.159	0.144	0.175	0.098	0.138	0.267	0.246	0.236	0.246	0.135	0.145
DynaDepth_R18	Mono	110.38	81.50	0.117	0.128	0.289	0.156	0.289	0.509	0.208	0.501	0.347	0.305	0.207	0.127	0.186	0.379	0.379	0.336	0.379	0.141	0.180
DynaDepth_R50	Mono	119.99	77.98	0.113	0.128	0.298	0.152	0.324	0.549	0.201	0.532	0.454	0.318	0.218	0.125	0.197	0.418	0.437	0.382	0.448	0.153	0.216
RA-Depth_HRNet	Mono	112.73	78.79	0.096	0.113	0.314	0.127	0.239	0.413	0.165	0.499	0.368	0.378	0.214	0.122	0.178	0.423	0.403	0.402	0.455	0.175	0.192
TriDepth_single	Mono	109.26	81.56	0.117	0.131	0.300	0.188	0.338	0.498	0.265	0.268	0.301	0.212	0.190	0.126	0.199	0.418	0.438	0.380	0.438	0.142	0.205
Lite-Mono_Tiny	Mono	92.92	89.55	0.115	0.127	0.257	0.157	0.225	0.354	0.191	0.257	0.248	0.198	0.186	0.127	0.159	0.358	0.342	0.336	0.360	0.147	0.161
Lite-Mono_Tiny+HR	Mono	86.71	87.63	0.106	0.119	0.227	0.139	0.282	0.370	0.166	0.216	0.201	0.190	0.202	0.116	0.146	0.320	0.291	0.286	0.312	0.148	0.167
Lite-Mono_Small	Mono	100.34	84.67	0.115	0.127	0.251	0.162	0.251	0.430	0.238	0.353	0.282	0.246	0.204	0.128	0.161	0.350	0.336	0.319	0.356	0.154	0.164
Lite-Mono_Small+HR	Mono	89.90	86.05	0.105	0.119	0.263	0.139	0.263	0.436	0.167	0.188	0.181	0.193	0.214	0.117	0.147	0.366	0.354	0.327	0.355	0.152	0.157
Lite-Mono_Base	Mono	93.16	85.99	0.110	0.119	0.259	0.144	0.245	0.384	0.177	0.224	0.237	0.221	0.196	0.129	0.175	0.361	0.340	0.334	0.363	0.151	0.165
Lite-Mono_Base+HR	Mono	89.85	85.80	0.103	0.115	0.256	0.135	0.258	0.486	0.164	0.220	0.194	0.213	0.205	0.114	0.154	0.340	0.327	0.321	0.344	0.145	0.156
Lite-Mono_Large	Mono	90.75	85.54	0.102	0.110	0.227	0.126	0.255	0.433	0.149	0.222	0.225	0.220	0.192	0.121	0.148	0.363	0.348	0.329	0.362	0.160	0.184
Lite-Mono_Large+HR	Mono	92.01	83.90	0.096	0.112	0.241	0.122	0.280	0.482	0.141	0.193	0.194	0.213	0.222	0.108	0.140	0.403	0.404	0.365	0.407	0.139	0.182

MonoDepth2_R18	Stereo	117.69	79.05	0.123	0.133	0.348	0.161	0.305	0.515	0.234	0.390	0.332	0.264	0.209	0.135	0.200	0.492	0.509	0.463	0.493	0.144	0.194
MonoDepth2_R18+nopt	Stereo	128.98	79.20	0.150	0.181	0.422	0.292	0.352	0.435	0.342	0.266	0.232	0.217	0.229	0.156	0.236	0.539	0.564	0.521	0.556	0.164	0.178
MonoDepth2_R18+HR	Stereo	111.46	81.65	0.117	0.132	0.285	0.167	0.356	0.529	0.238	0.432	0.312	0.279	0.246	0.130	0.206	0.343	0.343	0.322	0.344	0.150	0.209
DepthHints	Stereo	111.41	80.08	0.113	0.124	0.310	0.137	0.321	0.515	0.164	0.350	0.410	0.263	0.196	0.130	0.192	0.440	0.447	0.412	0.455	0.157	0.192
DepthHints_HR	Stereo	112.02	79.53	0.104	0.122	0.282	0.141	0.317	0.480	0.180	0.459	0.363	0.320	0.262	0.118	0.183	0.397	0.421	0.380	0.424	0.141	0.183
DepthHints_HR+nopt	Stereo	141.61	73.18	0.134	0.173	0.476	0.301	0.374	0.463	0.393	0.357	0.289	0.241	0.231	0.142	0.247	0.613	0.658	0.599	0.692	0.152	0.191

MonoDepth2_R18	M+S	124.31	75.36	0.116	0.127	0.404	0.150	0.295	0.536	0.199	0.447	0.346	0.283	0.204	0.128	0.203	0.577	0.605	0.561	0.629	0.136	0.179
MonoDepth2_R18+nopt	M+S	136.25	76.72	0.146	0.193	0.460	0.328	0.421	0.428	0.440	0.228	0.221	0.216	0.230	0.153	0.229	0.570	0.596	0.549	0.606	0.161	0.177
MonoDepth2_R18+HR	M+S	106.06	82.44	0.114	0.129	0.376	0.155	0.271	0.582	0.214	0.393	0.257	0.230	0.232	0.123	0.215	0.326	0.352	0.317	0.344	0.138	0.198
CADepth	M+S	118.29	76.68	0.110	0.123	0.357	0.137	0.311	0.556	0.169	0.338	0.412	0.260	0.193	0.126	0.186	0.546	0.559	0.524	0.582	0.145	0.192
MonoViT_MPViT	M+S	75.39	90.39	0.098	0.104	0.245	0.122	0.213	0.215	0.131	0.179	0.184	0.161	0.168	0.112	0.147	0.277	0.257	0.242	0.260	0.147	0.144
MonoViT_MPViT+HR	M+S	74.95	89.72	0.094	0.102	0.238	0.114	0.225	0.269	0.117	0.145	0.171	0.145	0.184	0.108	0.145	0.302	0.277	0.259	0.285	0.135	0.148

NYUDepth2-C

Model	mCE (%)	mRR (%)	Clean	Bright	Dark	Contrast	Defocus	Glass	Motion	Zoom	Elastic	Quant	Gaussian	Impulse	Shot	ISO	Pixelate	JPEG
BTS_R50	122.78	80.63	0.122	0.149	0.269	0.265	0.337	0.262	0.231	0.372	0.182	0.180	0.442	0.512	0.392	0.474	0.139	0.175
AdaBins_R50	134.69	81.62	0.158	0.179	0.293	0.289	0.339	0.280	0.245	0.390	0.204	0.216	0.458	0.519	0.401	0.481	0.186	0.211
AdaBins_EfficientB5^⭐	100.00	85.83	0.112	0.132	0.194	0.212	0.235	0.206	0.184	0.384	0.153	0.151	0.390	0.374	0.294	0.380	0.124	0.154
DPT_ViT-B	83.22	95.25	0.136	0.135	0.182	0.180	0.154	0.166	0.155	0.232	0.139	0.165	0.200	0.213	0.191	0.199	0.171	0.174
SimIPU_{R50+no_pt}	200.17	92.52	0.372	0.388	0.427	0.448	0.416	0.401	0.400	0.433	0.381	0.391	0.465	0.471	0.450	0.461	0.375	0.378
SimIPU_R50+imagenet	163.06	85.01	0.244	0.269	0.370	0.376	0.377	0.337	0.324	0.422	0.306	0.289	0.445	0.463	0.414	0.449	0.247	0.272
SimIPU_R50+kitti	173.78	91.64	0.312	0.326	0.373	0.406	0.360	0.333	0.335	0.386	0.316	0.333	0.432	0.442	0.422	0.443	0.314	0.322
SimIPU_R50+waymo	159.46	85.73	0.243	0.269	0.348	0.398	0.380	0.327	0.313	0.405	0.256	0.287	0.439	0.461	0.416	0.455	0.246	0.265
DepthFormer_{SwinT_w7_1k}	106.34	87.25	0.125	0.147	0.279	0.235	0.220	0.260	0.191	0.300	0.175	0.192	0.294	0.321	0.289	0.305	0.161	0.179
DepthFormer_{SwinT_w7_22k}	63.47	94.19	0.086	0.099	0.150	0.123	0.127	0.172	0.119	0.237	0.112	0.119	0.159	0.156	0.148	0.157	0.101	0.108

特性分析

For more detailed benchmarking results and to access the pretrained weights used in robustness evaluation, kindly refer to 实验结果.md.

生成"损坏"数据

You can manage to create your own "RoboDepth" corrpution sets! Follow the instructions listed in 数据生成.md.

更新计划

Initial release. 🚀
Add scripts for creating common corruptions.
Add download link of KITTI-C and NYUDepth2-C.
Add competition data.
Add benchmarking results.
Add evaluation scripts on corruption sets.

引用

If you find this work helpful, please kindly consider citing our paper:

@article{kong2023robodepth,
  title = {RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptions},
  author = {Kong, Lingdong and Xie, Shaoyuan and Hu, Hanjiang and Cottereau, Benoit and Ng, Lai Xing and Ooi, Wei Tsang},
  journal = {arXiv preprint arXiv:23xx.xxxxx}, 
  year = {2023},
}

@misc{kong2023robodepth_benchmark,
  title = {The RoboDepth Benchmark for Robust Out-of-Distribution Depth Estimation under Corruptions},
  author = {Kong, Lingdong and Xie, Shaoyuan and Hu, Hanjiang and Cottereau, Benoit and Ng, Lai Xing and Ooi, Wei Tsang},
  howpublished = {\url{https://github.com/ldkong1205/RoboDepth}}, 
  year = {2023},
}

许可

This work is under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

致谢

This project is supported by DesCartes, a CNRS@CREATE program on Intelligent Modeling for Decision-Making in Critical Urban Systems.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README_CN.md

README_CN.md

RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptions

项目概览

版本更新

大纲

安装

数据准备

RoboDepth Benchmark

Competition @ ICRA 2023

开始实验

模型库

🚘 - 室外深度估计

🏠 - 室内深度估计

鲁棒性基线

KITTI-C

NYUDepth2-C

特性分析

生成"损坏"数据

更新计划

引用

许可

致谢

Files

README_CN.md

Latest commit

History

README_CN.md

File metadata and controls

RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptions

项目概览

版本更新

大纲

安装

数据准备

RoboDepth Benchmark

Competition @ ICRA 2023

开始实验

模型库

🚘 - 室外深度估计

🏠 - 室内深度估计

鲁棒性基线

KITTI-C

NYUDepth2-C

特性分析

生成"损坏"数据

更新计划

引用

许可

致谢