Scan Model Aligner

Annotation tool for retrieving and aligning models to a scan.

If you use this annotation tool, please cite:

@inproceedings{dai2017scannet,
    title={ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes},
    author={Dai, Angela and Chang, Angel X. and Savva, Manolis and Halber, Maciej and Funkhouser, Thomas and Nie{\ss}ner, Matthias},
    booktitle = {Proc. Computer Vision and Pattern Recognition (CVPR), IEEE},
    year = {2017}
}

Example: http://localhost:8010/scans/scan-model-aligner?modelId=nyuv2.bedroom_0114

1. Click on an object to retrieve and align a model to the object

2. A list of possible matches (based on category) will pop up. Select a model to use it.

3. Use keyboard shortcuts or mouse based manipulator to align the model to the object.

   Up/Down - Scale up/down

   Left/Right - Rotate left/right

4. After aligning a model you can select a model instance and press 'r' to replace it or 'delete'/'backspace' to remove it.

Improvements Needed

1. Improved retrieval
   1. Backend infrastructure to suggest matched models given selected segment of scan
2. Improved alignment
   1. Better initial alignment (e.g. provide correct rotation based on OBB)
   2. User feedback on overlap
   3. Show several views so user can visual judge overlap from multiple viewpoints
3. Placement using support
   1. Walls/floors/ceiling are either automatically computed or provided by humans
   2. Allow toggle of using support when placing objects

Storing and retrieving annotations

Annotations are stored in the annotations data (described in Segment-Annotations#annotations-table)

View list of annotations for scan model aligner: http://localhost:8010/scans/annotations/list?task=scan-model-align

Get one annotation with JSON format http://localhost:8010/scans/annotations/list?format=json&id=17167

Use ssc/export-scan-model-alignments.js to export alignments as json files.

NODE_BASE_URL=http://localhost:8010 ./export-scan-model-alignments.js --id bedroom_0065  --source nyuv2 -n 1 --output_dir annotations

Raw annotation format

The data field contains the stringified JSON of the raw annotation.

Each alignment annotation has the following fields:

• id: sceneId
• data: object with scene and placement data

Under data.scene.scene (some weird nesting here), the important fields are:

• objects: Array of objects in the scene Each object has the following important fields:
  • index: the index of object in the array
  • modelId: id of the model representing the object
  • parentIndex: the index of the support parent of the object (-1 if no support parent)
  • transform.data: 4x4 scene transform matrix of the model (as array in column order)
• placedModels: Association of annotated segment groups (segGroupIndex) to object indices (mInstIndex)

Example annotation:

[
  {
    "id": 17167,
    "appId": "ScanModelAligner.v1",
    "sessionId": "local-session",
    "workerId": "test",
    "itemId": "nyuv2.bedroom_0114",
    "condition": "test",
    "data": {
      "scene": {
        "format": "sceneState",
        "scene": {
          "up": { "x": 0, "y": 0, "z": 1 },     // Up vector for scene
          "front": { "x": 0, "y": -1, "z": 0 }, // Front vector for scene
          "unit": 1,  // Unit in meters that the scene is represented in
          "object": [
            {
              "modelId": "nyuv2.bedroom_0114",  // First object should be the scan
              "index": 0,
              "parentIndex": -1,
              "transform": {  // 4x4 transformation matrix taking model to scene space in column order
                "rows": 4,
                "cols": 4,
                "data": [1,0,0,0, 0,1,0,0, 0,0,1,0, 0,0,0,1]
              }
            },
            {
              "modelId": "wss.10c706f946fa0f3ea3056fa1e8da3997",
              "index": 1,
              "parentIndex": -1,
              "transform": {
                "rows": 4,
                "cols": 4,
                "data": [...]
              }
            },
            {
              "modelId": "wss.225da2a2b7a461e349edb0f98d2a2a29",
              "index": 2,
              "parentIndex": -1,
              "transform": {         
                "rows": 4,
                "cols": 4,
                "data": [...]                  
              }
            }
          ],
          "camera": [
            {
              "up": { "x": 0, "y": 0, "z": 1 },
              "position": { "x": 3.55, "y": -1.27, "z": 8.54 },
              "target": { "x": 2.94, "y": 2.63, "z": 1.34 },
              "isOrtho": false,
              "name": "current"
            }
          ]
        },
        "selected": []
      },
      "segmentType": "surfaces",
      "placedModels": [      // Association of annotated segment groups (segGroupIndex) to object indices (mInstIndex)
        {
          "segGroupIndex": 7,
          "label": "bookshelf",
          "mInstIndex": 2
        },
        {
          "segGroupIndex": 9,
          "label": "picture",
          "mInstIndex": 1
        }
      ],
      "timings": {
        "times": {
          "initial": 1500686096287,
          "scanLoad": {
            "start": 1500686096519,
            "end": 1500686104515,
            "duration": 7996
          },
          "annotatorReady": 1500686106789,
          "annotationSubmit": 1500686157143
        },
        "durations": {
          "scanLoad": 7996,
          "annotatorReady": 10502,
          "annotationSubmit": 60856
        }
      }
    },
    "status": null,
    "code": null,
    "created_at": "2017-07-22T01:15:57.000Z",
    "updated_at": "2017-07-22T01:15:57.000Z",
    "task": "scan-model-align",
    "type": "scan-model-align",
    "taskMode": null,
    "notes": null,
    "verified": 0
  }
]