ALTO support for OCR of video

[acpopat]

ALTO should support OCR of video efficiently.

This is a future-looking issue, not something we're likely to address immediately, but something to keep in mind as we drive progress of ALTO to be a suitable representation of all OCR output or ground truth in general, whether the source be scanned documents, scene text, screenshots, or video.

Video may require special consideration because the straightforward approach, having an ALTO record for the result of OCR of each frame, would be grossly inefficient since in most videos, OCR is present in only some portions, and text tends to persist over segments, either entirely or partially.

To track and drive this capability, this issue proposes that ALTO should represent the "ideal" OCR results of the attached video in much the way a human commentator would: by describing the overall text once, and representing dynamically the changing text-region boundaries in the moving scene in an efficient manner, e.g., by encoding differences in bounding boxes or by describing the motion parametrically.

Considering video may also drive discussion of the relative roles of layout representation and text-fragment representation, and of collection-level annotation (e.g., book or video or newspaper) and page-level annotation.

Relevant files:

• video of occluding object exposing moving to expose different parts of the text
• video of same text without occlusion, but, with field of view sliding along the text to expose all of it
• photo of text, with shadows on it
• photo of text, no shadows
• original digital document containing the text prior to printing
• text in the original digital document containing the text prior to printing

This issue will be considered fixed when the following has happened:

For the referenced two video files, represent the ideal OCR results (i.e., OCR groundtruth) efficiently using ALTO and attach the XML files to this issue.

[acpopat]

Hi All, as discussed in our last meeting, I think ALTO should consider the use case where text in video is being OCR'ed. Some common types of video text:

• Incidental text in the background (signs, storefronts, etc.)
• hard captions or subtitles, e.g., those burned into the pixels
• soft captions or subtitles, e.g., born-digital
• titles and credits, sometimes scrolling

[jukervin]

Video OCR stuff in Microsoft Azure:
https://azure.microsoft.com/en-us/blog/ocr-on-azure-media-analytics/
https://azure.microsoft.com/en-us/blog/announcing-video-ocr-public-preview-new-config/
https://azure.microsoft.com/en-us/blog/text-recognition-for-video-in-microsoft-video-indexer/
https://docs.microsoft.com/en-us/azure/cognitive-services/video-indexer/video-indexer-output-json-v2