Segment Annotation Data Model

Annotation data model

For a broader discussion of the SSTK annotation table, see Annotation Tables.

For the original Scan Annotation Pipeline developed for ScanNet, we used segment_annotations to store the segmentation information. For the updated Multiscan Annotation Pipeline that had more granular annotation directly on triangles, the segmentation information is stored directly in the annotations table.

Annotations Table

Annotation for each annotated model is stored in the 'annotations' table

id - Unique identifier for the annotation
appId - String indicating what app/version was used to generate this label
sessionId - String identifying the session id
workerId - String identifying the annotator
condition - String identifying the experiment condition
itemId - String identifying the item being annotated
task - Task name
type - Annotation type ('segment' is used for annotations that link to the segment_annotations table)
taskMode - Task mode ('new', 'fixup', 'coverage' for segment annotations)
data - JSON blob of statistics and overall information about the annotation
preview_data - Image blob
status - Status string indicating whether the annotation is 'verified', 'cleaned', 'rejected' (not good for some reason), 'ignore' (either old or test cases), or 'flagged' for reviewing. Unprocessed annotations have an empty or null status.
notes - Additional freeform notes
verified - Boolean flag indicating if the annotation was verified
code - Code for verifying updates to annotation
created_at - Original timestamp of creation of annotation
updated_at - Last update timestamp

Example

Example record retrieved from: http://localhost:8010/scans/segment-annotations/list?$cleaned&status=cleaned&id=16002&format=json

[
  {
    "id": 16002,  // Unique id for this annotation 
    "appId": "SegmentAnnotator.v3",   // id + version of the tool used to create the annotation
    "sessionId": "...", // session id of annotation task
    "workerId": "...",  // id of MTurk worker or user who did this annotation
    "itemId": "...",    // id of scan the annotation was for
    "condition": "...",  // condition for grouping set of annotations
    "data": { ... ,      // json object with information about the annotation
      "stats": {
        "annotatedVertices": 103682,            
        "unannotatedVertices": 102727,           
        "annotatedSegments": 538,
        "unannotatedSegments": 1185,
        "totalSegments": 1723,
        "totalVertices": 206409,
        "percentComplete": 50.23133681186383    // percent complete based on annotatedVertices/totalVertices
      },
      "timings": {
        "times": {                     // Raw times of when certain things happened (in ms)
          "initial": 1491114138507,        // Some time point when then annotation tool was loaded 
          "modelLoad": {
            "start": 1491114140102,        // when the scan started to load
            "end": 1491114166972,          // when the scan finished loading
            "duration": 26870              // how long the load took (in ms)
          },
          "annotatorReady": 1491114174068,   // when the annotator was ready (everything loaded)
          "annotationSubmit": 1491117342124  // when the annotation was submitted
        },
        "durations": {                  // Useful durations (in ms)
          "modelLoad": 26870,           // scan load duration
          "annotatorReady": 35561,      // annotation tool ready duration
          "annotationSubmit": 3203617   // annotation submit duration (how long annotator took)
        }
      }
    },
    "status": "cleaned",
    "code": null,
    "created_at": "2017-04-02T07:18:30.000Z",
    "updated_at": "2017-04-02T16:12:12.000Z",
    "task": "segment_annotation",
    "type": "segment",
    "taskMode": "new",
    "notes": "",
    "verified": 0,
    "label": "dinner table,floor,ceiling,ceiling,door,doorframe,kitchen cabinet,kitchen counter,kitchen counter,stair,ladder,beam,ceiling,wall,kitchen cabinet,hanging light,chair", // Aggregated field: all labels in this annotation
    "nlabels": 13,         // Aggregated field: Number of unique labels in this annotation
    "ninstances": 17       // Aggregated field: Number of object instances
  }

Field details

The data field is a stringified JSON with the following information for the segment labeling task:

   { 
      "metadata" {
        "startFrom": "latest",     // What did we start from "latest", "aggr", integer of specific annotation id
        "startAnnotations": [...]  // Actual set of ids we started from  
        "segmentType": "surfaces"  // What segmentation did we use         
      },   
      "stats": {
        "annotatedVertices": 103682,            
        "unannotatedVertices": 102727,           
        "annotatedSegments": 538,
        "unannotatedSegments": 1185,
        "totalSegments": 1723,
        "totalVertices": 206409,
        "percentComplete": 50.23133681186383    // percent complete based on annotatedVertices/totalVertices
      },
      "timings": {
        "times": {                     // Raw times of when certain things happened (in ms)
          "initial": 1491114138507,        // Some time point when then annotation tool was loaded 
          "modelLoad": {
            "start": 1491114140102,        // when the scan started to load
            "end": 1491114166972,          // when the scan finished loading
            "duration": 26870              // how long the load took (in ms)
          },
          "annotatorReady": 1491114174068,   // when the annotator was ready (everything loaded)
          "annotationSubmit": 1491117342124  // when the annotation was submitted
        },
        "durations": {                  // Useful durations (in ms)
          "modelLoad": 26870,           // scan load duration
          "annotatorReady": 35561,      // annotation tool ready duration
          "annotationSubmit": 3203617   // annotation submit duration (how long annotator took)
        }
      }
   }

The notes field is free form text. Will typically be comma separated set of following:

shift - Cleaned labels are shifted by one from original due to weird worker
badLabel - Bad label is removed
relabel - A label was spell corrected
replace - The annotation has been replaced by another due to fixup
dup - The annotation was duplicate
badAnnotation - The annotation was just bad
badWorker - The worker is bad
cleanedFloor - Removed some segments from floor
removedFloor - Removed floor
singleLargeLabel - Whole scene was labeled with one big label
potentialOutlier - Flagged as potential outlier based on average object class height
mislabel - Object is labeled with the wrong class label (manual note)
overextended - Label covers more than the object (manual note)
multipleObjects - Label covers more than one of the same type of object (manual note)
partialObject - Label covers a (very) small subset of the object (manual note)

Segment Annotations Table

Annotation for each segment is stored in the segment_annotations table

id - Unique identifier for the labeled segment
annId - Annotation id (foreign key connecting back to the annotations table)
appId - String indicating what app/version was used to generate this label (duplicated from annotations table)
sessionId - String identifying the session id (duplicated from annotations table)
workerId - String identifying the annotator (duplicated from annotations table)
condition - String identifying the experiment condition (duplicated from annotations table)
modelId - String identifying the model being annotated (duplicated from itemId in annotations table)
objectId - Index of object instance (counts up from 0 for each annId)
segments - JSON blob of segment
label - Label
labelType - Label type (typically 'category')
status - Status string indicating whether the labeled segment is to be rejected or not
notes - Additional freeform notes
verified - Boolean flag indicating if the annotation was verified
created_at - Original timestamp of creation of labeled segment
updated_at - Last update timestamp

Cleaned annotations are copied into segment_annotations_clean. In addition to the above fields, segment_annotations_clean also has the following fields

created_at_cleaned - timestamp the cleaned copy was made
updated_at_cleaned - Last update timestamp for the cleaned record

Provide feedback

Saved searches