The existing datasets for evaluating Visual Inertial Odometry (VIO) have boosted the research of autonomous agents, but they don’t meet the prosperous research of Augmented Reality (AR) or Mixed Reality (MR) given that they are not collected at real AR scenes and do not account for affecting factors of mobile devices. This paper presents the NEAR dataset, an AR oriented visual-inertial dataset collected with commodity handheld phones with ground truthrts. The dataset has a total of 113 sequences in 49 elaborately designed collection cases at two typical indoor scenes, i.e. the living area and the table area. It also covers plenty of setting adjustments for comparison, including the comparisons of different level textures, illuminations, motion patterns, camera settings and the difference between the rolling shutter and the global shutter.
If you wanna acknowledge our work, please cite
- Cheng Wang, Yu Zhao, Jiabin Guo, Ling Pei, Yue Wang, Haiwei Liu. NEAR: The NetEase AR Oriented Visual Inertial Dataset. [Accepted by ISMAR 2019].
The dataset link is here.
| case | environment | scene | lighting | motion pattern | special device setting | devices and link | preview |
|---|---|---|---|---|---|---|---|
| 1 | Easy[1] | Rich-textured desktop | uniform & stable | circle (slow+normal) | default[5] | iPhone 7, Huawei P20 |  |
| 2 | Easy | Rich-textured desktop | uniform & stable | hybrid | default | iPhone 7, Huawei P20 |  |
| 2_control[4] | Easy | Rich-textured desktop | uniform & stable | hybrid | default | iPhone 7, Huawei P20, MYNT |  |
| 3 | Easy | Rich-textured desktop | uniform & stable | circle (alwayse points to center) | default | iPhone 7, Huawei P20 |  |
| 3_control | Easy | Rich-textured desktop | uniform & stable | circle (alwayse points to center) | default | iPhone 7, Huawei P20, MYNT |  |
| 4 | Easy | Medium-textured desktop | uniform & stable | circle (slow+normal) | default | iPhone 7, Huawei P20 |  |
| 5 | Easy | Medium-textured desktop | uniform & stable | hybrid | default | iPhone 7, Huawei P20 |  |
| 6 | Easy | Medium-textured desktop with reflection | uniform & stable | circle (slow+normal) | default | iPhone 7, Huawei P20 |  |
| 7 | Easy | Medium-textured desktop with reflection | uniform & stable | hybrid | default | iPhone 7, Huawei P20 |  |
| 8 | Easy | Rich-textured livingroom | uniform & stable | translation (slow+normal) | default | iPhone 7, Huawei P20 |  |
| 9 | Easy | Rich-textured livingroom | uniform & stable | circle (slow+normal) | default | iPhoneXR_OnePlus5T_part1 iPhoneXR_OnePlus5T_part2 |
 |
| 9_control | Easy | Rich-textured livingroom | uniform & stable | circle (slow+normal) | default | iPhoneXR_OnePlus5T_MYNT_part1 iPhoneXR_OnePlus5T_MYNT_part2 |
 |
| 10 | Easy | Rich-textured livingroom | uniform & stable | hybrid | default | iPhone XR, OnePlus 5T |  |
| 10_control | Easy | Rich-textured livingroom | uniform & stable | hybrid | default | iPhone XR, OnePlus 5T, MYNT |  |
| 11 | Easy | Rich-textured livingroom | uniform & stable | user scan simulation | default | iPhoneXR_OnePlus5T_part1 iPhoneXR_OnePlus5T_part2 |
 |
| 11_control | Easy | Rich-textured livingroom | uniform & stable | user scan simulation | default | iPhoneXR_OnePlus5T_MYNT_part1 iPhoneXR_OnePlus5T_MYNT_part2 |
 |
| 12 | Normal[2] | Rich-textured desktop | uniform & stable | circle (slow+normal+fast) | default | iPhone XR, OnePlus 5T |  |
| 13 | Normal | Rich-textured desktop | uniform & stable | hybrid | default | iPhone XR, OnePlus 5T |  |
| 14 | Normal | Rich-textured desktop | uniform & stable | free | default | iPhone XR, OnePlus 5T |  |
| 15 | Normal | Rich-textured desktop | uniform & stable | game simulation (YuMe) | default | iPhone XR, OnePlus 5T |  |
| 16 | Normal | Medium-textured desktop | uniform & stable | circle (slow+normal+fast) | default | iPhone XR, OnePlus 5T |  |
| 17 | Normal | Medium-textured desktop | uniform & stable | hybrid | default | iPhone XR, OnePlus 5T |  |
| 18 | Normal | Medium-textured desktop | uniform & stable | free | default | iPhone XR, OnePlus 5T |  |
| 19 | Normal | Medium-textured desktop | uniform & stable | game simulation (YuMe) | default | iPhone XR, OnePlus 5T |  |
| 20 | Normal | Medium-textured desktop | dynamic | circle (slow+normal+fast) | default | iPhone XR, OnePlus 5T |  |
| 21 | Normal | Medium-textured desktop | dynamic | hybrid | default | iPhone 7, Huawei P20 |  |
| 22 | Normal | Medium-textured desktop | dynamic | free | default | iPhone 7, Huawei P20 |  |
| 23 | Normal | Medium-textured desktop with reflection | uniform & stable | circle (slow+normal+fast) | default | iPhone 7, Huawei P20 |  |
| 24 | Normal | Medium-textured desktop with reflection | uniform & stable | hybrid | default | iPhone 7, Huawei P20 |  |
| 25 | Normal | Medium-textured desktop with reflection | uniform & stable | free | default | iPhone 7, Huawei P20 |  |
| 26 | Normal | Median-textured livingroom | uniform & stable | translation (slow+normal+fast) | default | iPhone 7, Huawei P20 |  |
| 27 | Normal | Median-textured livingroom | uniform & stable | translation (slow+normal+fast) | 30fps | iPhone 7, Huawei P20 |  |
| 28 | Normal | Median-textured livingroom | uniform & stable | translation (slow+normal+fast) | autofocus | iPhone 7, Huawei P20 |  |
| 29 | Normal | Median-textured livingroom | uniform & stable | circle (slow) | default | iPhone XR, OnePlus 5T |  |
| 30 | Normal | Median-textured livingroom | uniform & stable | circle (normal) | default | iPhone XR, OnePlus 5T |  |
| 31 | Normal | Median-textured livingroom | uniform & stable | circle (fast) | default | iPhone XR, OnePlus 5T |  |
| 32 | Normal | Median-textured livingroom | uniform & stable | circle (normal) | 30fps | iPhone XR, OnePlus 5T |  |
| 33 | Normal | Median-textured livingroom | uniform & stable | circle (normal) | autofocus | iPhone XR, OnePlus 5T |  |
| 34 | Normal | Median-textured livingroom | uniform & stable | hybrid | default | iPhone XR, OnePlus 5T |  |
| 35 | Normal | Median-textured livingroom | uniform & stable | hybrid | 30fps | iPhone XR, OnePlus 5T |  |
| 36 | Normal | Median-textured livingroom | uniform & stable | hybrid | autofocus | iPhone XR, OnePlus 5T |  |
| 37 | Normal | Median-textured livingroom | uniform & stable | free | default | iPhone XR, OnePlus 5T |  |
| 38 | Normal | Median-textured livingroom | uniform & stable | user scan simulation | default | iPhone XR, OnePlus 5T |  |
| 39 | Normal | Median-textured livingroom | uniform & stable | game simulation (Shot) | default | iPhone 7, Huawei P20 |  |
| 40 | Normal | textureless floor and furnitures | uniform & stable | translation (slow+normal+fast) | default | iPhone 7, Huawei P20 |  |
| 41 | Normal | textureless floor and furnitures | uniform & stable | circle (slow) | default | iPhone 7, Huawei P20 |  |
| 42 | Normal | textureless floor and furnitures | uniform & stable | circle (normal) | default | iPhone 7, Huawei P20 |  |
| 43 | Normal | textureless floor and furnitures | uniform & stable | circle (fast) | default | iPhone 7, Huawei P20 |  |
| 44 | Normal | textureless floor and furnitures | uniform & stable | hybrid | default | iPhone 7, Huawei P20 |  |
| 45 | Normal | textureless floor and furnitures | uniform & stable | free | default | iPhone 7, Huawei P20 |  |
| 46 | Hard[3] | Rich-textured desktop | non-uniform & stable | hybrid | default | iPhone 7, Huawei P20 |  |
| 48 | Hard | Medium-textured desktop | non-uniform & stable | hybrid | default | iPhone 7, Huawei P20 |  |
| 50 | Hard | Medium-textured desktop with reflection | non-uniform & stable | hybrid | default | iPhone 7, Huawei P20 |  |
| 52 | Hard | textureless floor and furnitures | non-uniform & stable | hybrid | default | iPhone 7, Huawei P20 |  |
notes:
- "Easy" means no repetitive texture, no reflection, no opened windows. Rich-textured carpets, sofas and walls.
- "Normal" means a common livingroom settings. Median-textured carpets, sofas and walls. Some opened windows and a mirror.
- "Hard" means Opened windows, mirrors, textureless floor, textureless furnitures, textureless walls, lamps.
- "_control" means we have an additinal MYNT camera attached to record data besides of the two mobile phones.
- default device setting means as below.
The full list of the case design can be downloaded with CaseTableEva.xlsx and the better resolution preview video can be found at NEAR-dataset-preview-video.
For full understand our design, please refer to our paper[1].
We tested some open source VIOs including ICE-BA, VINS-Mono, OKVIS, ROVIO, MSCKF-Mono on our NEAR dataset tentatively. Note that the results are not for benchmarking.
Based on the results, we devide our dataset into several groups as below for reference.
The sequences are organized following the structure of the EuRoC, so the usage is fair easy.
./Case ID
│
└───ip7 A folder with phone name.
└───P_lp2lv.txt The extrinsic between phone camera and rigid body.
└───rts_compensate_extrinsic_and_time.txt The ground truth trajectory (TUM format) of camera frame.
└───timeoffset.txt Timestamp offset between phone camera and RTS system.
│
└───mav0
│
└───cam0
│ └───data The folder contains image sequence.
│ └───cam.yaml Camera settings when record this sequence.
│ └───data.csv Timestamps of camera frames.
│ └───sensor.csv Camera intrinsics and camera-IMU extrinsic.
└───imu0
└───data.csv Timestamps and IMU measurements.
└───sensor.yaml IMU intrinsics and IMU-rig extrinsic which is identity.