Discussion on Trick Play / Scrubbing Preview Images

[nicknsy]

Hi, I wanted to consolidate some of the discussion from this thread from jellyfin-roku as well as this thread from my own plugin repository. I would love to be able to contribute to jellyfin by working on an official trick play functionality, but obviously that requires work on every different client to accomplish, so this is mainly about trying to find a format that can work for all platform developers or at least seeing what platforms would be the easiest to compromise on. From the research I've done:

Formats:

BIF files
A standard originally made by Roku -- essentially just a stream of jpgs

Pros:

• Easily created/decoded without need for third-party tools (besides already bundled ffmpeg)
• Relatively easy to create client code to decode and use stream of jpgs
• Used by Netflix, Disney+, Plex among others
• Single network request to retrieve trick play data for an entire stream
• Easily stored and moved

Cons:

• Only natively supported by Roku, needs specific implementation on Swiftin (or any other native players in the future)

WebVTT
Standard HTML web element (I think)

Pros:

• Natively exists in HTML
• Implemented with grid of images (ex: https://i.imgur.com/7jfxvaQ.png), so still relatively easy to store
• May require third-party tools to create? (e.g. ffmpeg + imagemagick or media-ingestion)

Cons:

• Only HTML standard? May require additional implementation for native players like Roku, Swiftin
• Grid of images may make that implementation less simple

HLS
Standard made by Apple for video streaming.

Pros:

• Natively supported by Roku and tvOS

Cons:

• Different* implementations of trick play for both Roku and tvOS
• HLS.js doesn't support any trick play functionality so would also require custom impl

Clients:

tvOS/Swiftin
From what I can find on tvOS, it only natively supports using the HLS tag #EXT-X-I-FRAMES-ONLY which is an additional playlist with segments of individual frames that can be used for scrubbing. The only example I could find of this is here, where each individual preview image is its own .ts transport stream. I'll admit that all of HLS is a bit confusing to me, but if each frame would need to be its own transport stream I see that being a potentially big hassle to implement for static content and for clients that don't natively support scrubbing with #EXT-X-I-FRAMES-ONLY.

Can custom UI elements be used instead for scrubbing to support more formats?

Roku
Natively supports BIF. On their page for "trick mode" it also says SceneGraph (?) natively supports HLS. However, Roku claims on this page that Apple's approach can be less than ideal at times: "The format currently specified by HLS is a coded video I-frame. This approach is efficient because it references frame data directly from the video stream and has the added benefit of supporting digital rights management. However, additional video decoding resources are required to process these I-frames for display, which can be problematic in some cases. To make these images more accessible on a wider range of client devices, Roku supports a second playlist format, which it developed in collaboration with Disney and WarnerMedia". As such, they also support #EXT-X-TILES which are individual jpeg images that are grids containing multiple frames to be used for scrubbing. Theoretically, this means it could be interchangeably used with the images used by WebVTT.

Can custom UI elements be used instead for scrubbing to support more formats?

Web (also currently desktop + mobile)
Easiest to work with, and already has page elements for trick play images. Currently, I've already made support for BIFs but I could see WebVTT working. HLS.js (which I believe jellyfin-web uses) does not support #EXT-X-I-FRAMES-ONLY, and naturally only Roku supports #EXT-X-TILES. I'm really not sure how a custom implementation using #EXT-X-I-FRAMES-ONLY would be accomplished on the web client or on the server side as far as extracting frames goes.

[nielsvanvelzen]

I've just checked how we'd implement such a thing in the Android apps and it looks like we'll need to write custom code for it anyway, so there is no real preference for the format. I think the BIF files should work fine.

It would be nice if the server can expose two data points for video streams;

1. Do we have scrubbing previews?
2. Where can we find the previews?

This can likely be done by adding something like "ScrubbingPreviewsKey" to the stream, when null there is no preview and otherwise there is and we can get it by requesting a certain (authenticated!) endpoint with that key which in turn will return the BIF/other.

[iwalton3]

For anyone following this, I created an issue here to discuss implementation of HLS tile generation as a server feature to upstream: nicknsy/jellyscrub#83 (comment)

[justjanne]

In the long term, it would be awesome if it would be possible to pre-generate the trickplay images (and metadata) in some format and store it alongside the content.

In my case I encode all my media into HLS/DASH, store them on S3, and import them via .strm files. Storing the media as static files allows me to drastically reduce the load on jellyfin.

Either support for importing trickplay from .strm files referencing HLS streams, or support for storing the metadata alongside the .strm file in the media library would be awesome.

[sevenrats]

second for placing out-of-band images in media files.

[Maxplosion]

please remember there are users with read-only filesystems where only the jellyfin conf dir is writeable. so a settings switch on where to store the files would be appreciated.