diff --git a/crates/store/re_types/definitions/rerun/archetypes/asset_video.fbs b/crates/store/re_types/definitions/rerun/archetypes/asset_video.fbs index 29bc6bf234b6..0bb4f4ec979c 100644 --- a/crates/store/re_types/definitions/rerun/archetypes/asset_video.fbs +++ b/crates/store/re_types/definitions/rerun/archetypes/asset_video.fbs @@ -4,9 +4,10 @@ namespace rerun.archetypes; /// /// NOTE: Videos can only be viewed in the Rerun web viewer. /// Only MP4 containers with a limited number of codecs are currently supported, and not in all browsers. -/// Follow for updates on the native support. /// -/// In order to display a video, you need to log a [archetypes.VideoFrameReference] for each frame. +/// See for details of what is and isn't supported. +/// +/// In order to display a video, you also need to log a [archetypes.VideoFrameReference] for each frame. /// /// \example archetypes/video_auto_frames title="Video with automatically determined frames" image="https://static.rerun.io/video_manual_frames/320a44e1e06b8b3a3161ecbbeae3e04d1ccb9589/1200w.png" /// \example archetypes/video_manual_frames title="Demonstrates manual use of video frame references" image="https://static.rerun.io/video_manual_frames/9f41c00f84a98cc3f26875fba7c1d2fa2bad7151/1200w.png" diff --git a/crates/store/re_types/src/archetypes/asset_video.rs b/crates/store/re_types/src/archetypes/asset_video.rs index 1985f1b62ad6..5070dd5ac589 100644 --- a/crates/store/re_types/src/archetypes/asset_video.rs +++ b/crates/store/re_types/src/archetypes/asset_video.rs @@ -22,9 +22,10 @@ use ::re_types_core::{DeserializationError, DeserializationResult}; /// /// NOTE: Videos can only be viewed in the Rerun web viewer. /// Only MP4 containers with a limited number of codecs are currently supported, and not in all browsers. -/// Follow for updates on the native support. /// -/// In order to display a video, you need to log a [`archetypes::VideoFrameReference`][crate::archetypes::VideoFrameReference] for each frame. +/// See for details of what is and isn't supported. +/// +/// In order to display a video, you also need to log a [`archetypes::VideoFrameReference`][crate::archetypes::VideoFrameReference] for each frame. /// /// ⚠️ **This type is experimental and may be removed in future versions** /// diff --git a/docs/content/reference/types/archetypes/asset_video.md b/docs/content/reference/types/archetypes/asset_video.md index d7d67b6af057..920b1a9cce5f 100644 --- a/docs/content/reference/types/archetypes/asset_video.md +++ b/docs/content/reference/types/archetypes/asset_video.md @@ -10,9 +10,10 @@ A video binary. NOTE: Videos can only be viewed in the Rerun web viewer. Only MP4 containers with a limited number of codecs are currently supported, and not in all browsers. -Follow for updates on the native support. -In order to display a video, you need to log a [`archetypes.VideoFrameReference`](https://rerun.io/docs/reference/types/archetypes/video_frame_reference?speculative-link) for each frame. +See for details of what is and isn't supported. + +In order to display a video, you also need to log a [`archetypes.VideoFrameReference`](https://rerun.io/docs/reference/types/archetypes/video_frame_reference?speculative-link) for each frame. ## Components diff --git a/docs/content/reference/video.md b/docs/content/reference/video.md new file mode 100644 index 000000000000..660f71cfa096 --- /dev/null +++ b/docs/content/reference/video.md @@ -0,0 +1,73 @@ +--- +title: Video +order: 2000 +--- + +## Video in Rerun +A stream of images (like those produced by a camera) can be logged to Rerun in several different ways: + +* Uncompressed, as many [`Image`](../reference/types/archetypes/image.md)s +* Compressed as many [`EncodedImage`](../reference/types/archetypes/encoded_image.md)s, using e.g. JPEG-encoding +* Compressed as a single [`AssetVideo`](../reference/types/archetypes/asset_video.md), using e.g. MP4 and AV1 or H.264. + +These alternatives range on a scale of "simple, lossless, and big" to "complex, lossy, and small". + +If you want lossless encoded images (with no compression artifacts), then you should log each video frame as `Image`. +This will use up a lot of space and bandwidth. You can also encode them as PNG and log them as `EncodedImage`, +though it should be noted that PNG encoding usually does very little for the file size of photographic images. + +If you want to reduce bandwidth and storage cost, you can encode each frame as a JPEG and log it using `EncodedImage`. This can easily reduce the file sizes by almost two orders of magnitude with minimal perceptiall loss. +This is also very simple to do, and the Python logging SDK has built-in support for it using [`Image.compress`](https://ref.rerun.io/docs/python/0.18.2/common/archetypes/#rerun.archetypes.Image.compress). + +Finally, you can encode the images as a video file, and log it using `AssetVideo`. +This gives the best compression ratio, reducing file sizes and bandwidth requirements. + +## `AssetVideo` limitations +Video support is new in Rerun, and has several limitations: + +* Video playback only works in the web viewer +* There is no video encoder in the Rerun SDK, so you need to create the video file yourself +* Only the MP4 container format is supported +* Only a limited sets of codecs are supported (see below) +* There is no audio support + +## Web viewer support +As of writing, playback of `AssetVideo` is only supported on the web viewer. +Native video playback is coming, and can be tracked [in this GitHub issue](https://github.com/rerun-io/rerun/issues/7298). + +Video playback is done using the browsers own video decoder, so the supported codecs depends on your browser. + +Overall, we recommend using Chrome or another Chromium-based browser, as it seem to have the best video support as of writing. + +When choosing codec, we recommend [AV1](https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Video_codecs#av1), as that seems to have the best overall playback support. Since AV1 is patent-free, it is also likely the first codec we will support in the native viewer. + +For decoding video in the Web Viewer, we use the [WebCodecs API](https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_API). +This API enables us to take advantage of the browser's hardware accelerated video decoding capabilities. +It is implemented by all modern browsers, but with varying levels of support for different codecs, and varying levels of quality. + +With that in mind, here are the browsers which we have tested and verified to work: + +| | Linux | macOS | Windows | +| ---------- | ------ | ----- | ------- | +| Firefox | ✅[^2] | ✅ | ✅ | +| Chrome[^1] | ✅ | ✅ | ✅ | +| Safari | | ✅ | | + +When it comes to codecs, we aim to support any codec which the browser supports, but +we currently cannot guarantee that all of them will work. For more information about +which codecs are supported by which browser, see [Video codecs on MDN](https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Video_codecs#codec_details). + +At the moment, we test the following codecs: + +| | Linux | macOS | Windows | +| ----- | ----- | ----- | ------- | +| AV1 | ✅ | 🚧\* | ✅ | +| H.264 | ✅ | ✅ | ✅ | + +* AV1 on macOS is only supported on [Apple Silicon devices with AV1 hardware decoding support](https://webkit.org/blog/14445/webkit-features-in-safari-17-0/). + +[^1]: Any Chromium-based browser should work, but we don't test all of them. +[^2]: Firefox on Linux has been observed to [stutter when playing back H.264 video](https://github.com/rerun-io/rerun/issues/7532) + +## Links +* [Web video codec guide, by Mozilla](https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Video_codecs) diff --git a/rerun_cpp/src/rerun/archetypes/asset_video.hpp b/rerun_cpp/src/rerun/archetypes/asset_video.hpp index 591f0ae904cc..03ea516a4ed6 100644 --- a/rerun_cpp/src/rerun/archetypes/asset_video.hpp +++ b/rerun_cpp/src/rerun/archetypes/asset_video.hpp @@ -23,9 +23,10 @@ namespace rerun::archetypes { /// /// NOTE: Videos can only be viewed in the Rerun web viewer. /// Only MP4 containers with a limited number of codecs are currently supported, and not in all browsers. - /// Follow for updates on the native support. /// - /// In order to display a video, you need to log a `archetypes::VideoFrameReference` for each frame. + /// See for details of what is and isn't supported. + /// + /// In order to display a video, you also need to log a `archetypes::VideoFrameReference` for each frame. /// /// ## Examples /// diff --git a/rerun_py/rerun_sdk/rerun/archetypes/asset_video.py b/rerun_py/rerun_sdk/rerun/archetypes/asset_video.py index 9ff82fc9571f..0499a6ab4792 100644 --- a/rerun_py/rerun_sdk/rerun/archetypes/asset_video.py +++ b/rerun_py/rerun_sdk/rerun/archetypes/asset_video.py @@ -23,9 +23,10 @@ class AssetVideo(AssetVideoExt, Archetype): NOTE: Videos can only be viewed in the Rerun web viewer. Only MP4 containers with a limited number of codecs are currently supported, and not in all browsers. - Follow for updates on the native support. - In order to display a video, you need to log a [`archetypes.VideoFrameReference`][rerun.archetypes.VideoFrameReference] for each frame. + See for details of what is and isn't supported. + + In order to display a video, you also need to log a [`archetypes.VideoFrameReference`][rerun.archetypes.VideoFrameReference] for each frame. ⚠️ **This is an experimental API! It is not fully supported, and is likely to change significantly in future versions.**