LayoutNet

New: Please check our official PyTorch implementation for LayoutNet v2

Torch implementation of our CVPR 18 paper: "LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image"

See sample video of 3D reconstruced layouts by our method.

Prerequisites

• Linux
• NVIDIA GPU + CUDA CuDNN
• Torch 7

matio: https://github.com/tbeu/matio

• Matlab

Data

• Download preprocessed (aligned to horizontal floor plane) training/validation/testing data to current folder

This includes the panoramas from both the panoContext dataset and our labeled stanford 2d-3d dataset.

• Download groundtruth data to current folder

This includes the groundtruth 2D position of room corners in .mat format from the two dataset. We've corrected some wrong corner labels in PanoContext to match the layout boundaries.

• Download preprocessed LSUN training/validation/testing data and related .t7 file under /data/LSUN_data/ folder. We've corrected 10% wrong corner labels.

• We provide training/tesing .t7 samples (selected from PanoContext) to train for general Manhattan layouts prediction. A few sub-samples have non-cuboid room shape ground truth.

Pretrained model

• Download our pretrained model to current folder. This includes:

1. The pretrained full approach on the panoContext dataset, the joint boudary and corner prediction branch, the single boundary prediction branch and the 3D layout box regressor;

2. The pretrained full approach on the LSUN dataset (we've corrected 10% wrong labels), the joint boudary and corner prediction branch and the single boundary prediction branch.

• The pretrained model for non-cuboid room shape prediction on the panoContext dataset. Using our labeled non-cuboid and cuboid room shape data.

Image preprocess

We provide sample script to extract Manhattan lines and align the panorama in ./matlab/getManhattanAndAlign.m.

To get gt edge map, corner map and box parameters, see sample script ./matlab/preprocessPano.m

To convert gt data to .t7 file, see sample code preProcess_pano.lua

Train network

• To train our full approach:

th driver_pano_full.lua

Note that this loads the pretrained joint prediction branch and the 3D layout box regressor.

• To train the joint prediction branch of boudary and corner:

th driver_pano_joint.lua

Note that this loads the pretrained boundary prediction branch.

• To train the boudary prediction branch:

th driver_pano_edg.lua

• To train the layout box regressor:

th driver_pano_box.lua

Test network

• To test on our full approach:

th testNet_pano_full.lua

This saves predicted boundary, corner and 3D layout parameter in "result/" folder.

Optimization

• To Add Manhattan constraints and optimize for a better layout, open Matlab, then:

cd matlab
panoOptimization.m

This loads saved predictions from the network output and performs sampling.

Evaluation

We provide the Matlab evaluation code for 3D IoU (compute3dOcc_eval.m) and the generation of 2D layout label (getSegMask_eval.m) for evaluating layout pixel accuracy.

Extension to perspective images

• To train our full approach:

th driver_persp_joint_lsun_type.lua

Note that this loads the pretrained joint corner and boundary prediction branch.

• To train the joint prediction branch of boudary and corner:

th driver_persp_joint_lsun.lua

Note that this loads the pretrained boundary prediction branch.

• To train the boudary prediction branch:

th driver_persp_lsun.lua

• To test the trained network:

th testNet_persp_full_lsun.lua

Note that this saves predicted boundary, corner and room type in "result/" folder. To get the exact 2D corner position on the image, run the following using Matlab:

cd matlab
getLSUNRes.m

You need to download the LSUN data and the toolbox to run through the experiment.

Miscellaneous

• To get reference for the labeling tool please check panoLabelTool.m.

Citation

Please cite our paper for any purpose of usage.

@inproceedings{zou2018layoutnet,
  title={LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image},
  author={Zou, Chuhang and Colburn, Alex and Shan, Qi and Hoiem, Derek},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2051--2059},
  year={2018}
}