Core Segment Skipping Proposal

[thornbill]

I think the path forward for adding intro skipping support to Jellyfin is to add it in a generic way such that it can be used for intro/outro skipping, commercial skipping, and potentially others.

1. Add support for skip lists to items in server. (List of start times, end times (or duration), and type. Basically what EDL uses + custom types for intros and outros.)

2. Clients check for skip times (and types) to automatically skip or show skip button.

3. Plugins can write to skip lists using whatever scanning etc they need.

A SkipSegments field should be added to the BaseItem using the following structure.

SkipSegments: [{
    StartPositionTicks: Ticks
    EndPositionTicks: Ticks
    Type: SkipType
}]

Where SkipType is an enum with the following values.

SkipType {
    SKIP
    MUTE
    SCENE_MARKER # or POI
    COMMERCIAL_BREAK
    INTRO
    OUTRO
}

Clients could then take action based on the SkipType at the proper time. Some skip types directly correspond to the action that should be performed, but others such as Scene Marker, Commercial Break, Intro, and Outro could potentially be configured per client to Skip, Prompt to Skip, or Mute.

Other Software Supported Skip Types/Actions

MPlayer EDL Actions

• 0 - Cut
• 1 - Mute
• 2 - Scene Marker (if start and end times are specified, the end time is used)
• 3 - Commercial Break

SponsorBlock Action Types

• skip
• mute
• full
• poi
• chapter

NOTE: SponsorBlock also has a category field that specifies the segment category (intro, outro, sponsor, selfpromo, etc.) separately from the action that should be taken.

Alternative Approaches

Using Chapters

Another proposed approach has been to reuse the existing Chapter support. This solution is insufficient in my opinion due to the following:

1. Chapters are a separate well defined concept in media.
2. Clients have existing behavior defined for Chapters that would not be applicable to Skip Segments.
3. Chapters have images and support automatic image extraction that is not relevant for Skip Segments.
4. Besides StartPositionTicks there is no overlap between the data required for Skip Segments vs Chapters

Chapters schema for reference:

Chapters: [{
    StartPositionTicks: Ticks
    Name: String
    ImagePath: String
    ImageDateModified: DateTime
    ImageTag: String
}]

References

• Kodi EDL — https://kodi.wiki/view/Edit_decision_list
• SponsorBlock Types — https://wiki.sponsor.ajay.app/w/Types
• SponsorBlock API — https://wiki.sponsor.ajay.app/w/API_Docs#GET_/api/skipSegments

[nielsvanvelzen]

Overall this proposal looks great. I believe keeping chapters & segments separate is the right approach. I got one question and a suggestion;

1. How does this work when I have a movie with 2 versions where some segments differ?
2. I think we don't need to add the value type as part of it's name, so "StartPositionTicks" could be "StartPosition" or just "Start".

SkipSegments: [{
    StartPosition: Ticks
    EndPosition: Ticks
    Type: SkipType
}]

or

SkipSegments: [{
    Start: Ticks
    End: Ticks
    Type: SkipType
}]

[thornbill]

1. I'm honestly not sure about this. Hopefully some server people can chime in with ideas on how to manage that. I don't think we support different Chapters per Media Source currently. 🤷‍♂️
2. Initially I had StartTime/EndTime but changed it to match what Chapters currently use. It looks like we are pretty consistent with using either StartPositionTicks or StartTimeTicks in the API currently.

[dkanada]

Current endpoints might be using Ticks in the names but since this is a new endpoint I think removing them would be the best option. We should be pushing in that direction for all endpoints when possible.

[neilsb]

I wonder if SkipType should be a simple classification of the content of the media, rather than also possibly being a recommended action (e.g. Skip or Mute), and then either have a system wide (or set at user preferences level) default action for that content type (e.g. None, Auto Skip, Prompt Skip, Auto Mute, Prompt Mute)? Rather than whatever is analyzing the media have to make a call on whether it should be skipped or muted.

And how detailed should SkipType values go. Intro, Outro and Commercial are the most common, but Recap would be useful for TV shows.

[thornbill]

I would agree with you except that EDL files are pretty limited in what they support. If we want to support that use case (and I think we should personally), I believe the generic Skip and Mute action types have to be supported, but the content/category types should be preferred.

I think Recap could make sense to add also.

[dkanada]

@thornbill mentioned SponsorBlock has Category to indicate the purpose of the segment and Type to indicate the suggested action, which sounds similar to what you're suggesting here, and I agree we should split them out.

[LePips]

These objects contextually represent "a period or a moment of time in media". Due to this, I think that just a single type can represent current Chapters and these "SkipSegments", however we don't need to constrain this to just "skips".

I propose the following encoding, names subject to change:

class MediaSegment {
	long StartPositionTicks
	long EndPositionTicks
    MediaSegmentType Type
	string Name
	string ImagePath
	DateTime ImageDateModified
	string ImageTag
}

Which allows and would result in:

• Segments of time to be defined within the range of [StartPositionTicks, EndPositionTicks]
• A single moment of time to be defined by just setting StartPositionTicks, leaving EndPositionTicks null [0]
• Any segment can have an image. I don't see a valid reason why not to have this, it adds more functionality
  • Optional for any type of segment
• Doesn't break existing Chapter support, expected encoded values are still provided
• Any MediaSegmentType can be defined, example:

enum MediaSegmentType {
    SKIP
    MUTE
    SCENE_MARKER // or POI
    COMMERCIAL_BREAK
    INTRO
    OUTRO
    CHAPTER
	RECAP
    // more
}

---

[0] - Chapters can be defined as a range instead of a single moment in time. The very context of chapters is "I am in a chapter and when I select a chapter, I want to go to the beginning of the chapter". Just like a book. So, technically, chapters should have EndPositionTicks (idk how ffmpeg parses, but we can determine the pairwise ranges and set accordingly). I think mkv even has chapters as ranges. Nonetheless, it's up to the client to define how they would handle this.

[thornbill]

I'm slowly coming around to this approach. Since we should be able to maintain compatibility with the current API pretty trivially just by returning Chapters in the current format in addition to the new MediaSegments.

I do think CHAPTER can replace SCENE_MARKER in the types. Reading more into the Kodi documentation they seem to be the same concept.

[LePips]

Thought about a possible, but not necessary, property to fulfill some use cases and allow more functionality.

Metadata Property

Some segment types might also use some kind of metadata. For example, assume the following additional property and case. Don't focus on the dictionary type for now, just the idea of it.

class MediaSegment {
    // original properties
    Dictionary<string, string> Metadata
}

enum MediaSegmentType {
    // original cases
    OTHER
}

Assume I have a plugin that wants the player to show some text on screen during a certain segment, like an image or relevant video information (like fun facts). I would be able to create a MediaSegment with relevant metadata, like:

• "LOTR_FUN_FACT_METADATA": "Did you know that Viggo Mortensen actually broke his toe when he kicked the helmet and the scream was real!?"
• "CHARACTER_PROFILE_URL_METADATA": "<image_url>"
• "CHARACTER_PROFILE_INFO_METADATA": "<information about this character>"

This means that "plugins can be used through the official API", in a sense. For clients, it probably wouldn't be much to support plugins and doesn't necessarily go against the "use the official API" rule we've recently made.

[dkanada]

It feels like this could be added with a POI category and optional Description and Link fields in the metadata, which would be preferable to a loose dictionary of custom entries, but that still sounds a bit excessive for this feature. Do we even have any possible sources to use as POI like that?

[LePips]

I think a loose dictionary would be fine, plugins would just have to document their keys. There can also hypothetically be more metadata than a Description and Link, I just provided that in the example so it shouldn't be restrictive.

There are currently no sources right now, however the idea is that there could be.

[dkanada]

Based on comments in here and the SponsorBlock documentation I think some new enums might look like this. We'd be missing the cut action from EDL but I'm not sure it would serve a purpose when the clients can't actually cut streams. Also added a preview category that doesn't seem to exist anywhere. Several categories and actions from SponsorBlock were excluded (full, filler, sponsor, promo, etc) because I couldn't find a good reason to add them yet.

enum MediaSegmentAction {
    NONE
    SKIP
    MUTE
}

enum MediaSegmentCategory {
    CHAPTER
    COMMERCIAL
    INTRO
    OUTRO
    PREVIEW
    RECAP
}

[LePips]

I don't think splitting to include an action enum would fit, a segment is representative of its type and it's purely up to the developer to do what they want during a given segment. While the action enum would "tell the client what to do during a segment", I think that's very much known by developers: "I want to provide skipping during the intro/etc.".

System actions (MUTE, BRIGHTNESS_DOWN, etc.) can still be a type, represented by either a range or an instant in time:

• range: during this range, perform a system action (MUTE)
• instant: at this instant, perform a system action (BRIGHTNESS_DOWN)

Of course, what one does outside of a range (like unmuting after a mute segment) is purely up to developer implementation.

While types like MUTE and BRIGHTNESS_DOWN seem like actions they can also be phrased as descriptors, and not one-time actions:

• MUTE: this is a muted segment
• BRIGHTNESS_DOWN: this is a segment where the brightness is lowered

[ajayyy]

The reason for splitting mute and skip in SponsorBlock is to be able to block out audio-only sponsor mentions, and improve transitions (mute a bit right before a skip if the audio starts early).

Only the second one really applies to intros, previews, recaps, etc.

[sevenrats]

Another potential application of skipping in Jellyfin-land would be skips that transform extended/collector's cuts into other, shorter cuts.
These skips would be different from others in that they would not show a button during playback, but would instead create selectable versions. This would allow the user to have a single all-inclusive cut of a film, but then define new versions of it based on scene skips. These could define theatrical cuts, for example, or family-friendly versions of films which skip inapprpriate content.

Expanding on this, it would be excellent if it we could exclude some tracks from skipping so that we could keep commentary or rifftrax that may only be for the theatrical cut. When defining a section to be skipped, we would need to option to indicate the tracks that were excluded from skipping. And of course, it would be awesome if, when selecting a track which was excluded by a set of skips, we either warn the user that its for a different version, or automatically apply the skips.

[meonkeys]

Also https://features.jellyfin.org/posts/1036/bookmark-audio-video-segments

Might help a bit with https://features.jellyfin.org/posts/2786/create-clips-from-media , too (although according to Venson in chat that would take more work: "...would require some deep integration with the webUI"

[JPVenson]

Also its a bit outside of the whole idea of media segments. Segments are persistant and not for transiant actions like creating clips. You dont want to create the clip you stored over and over again. @meonkeys

[0dragosh]

Is there anything that's holding this back? Would love to put a bounty on this if anyone is willing to take a crack at it.

[thornbill]

Not really... I had started a branch a while ago that started implementing the api for retrieving media segments as described here, but it still needs all the database work to store them and migrate the existing chapter data. The database work seemed a bit daunting for me since I'm not really a C# developer and then I got kind of busy with other things (mainly the Jellyfin web client and a personal project).

Just as a heads-up, the Jellyfin project is strongly opposed to bug/feature bounties so that really wouldn't be welcome.

[0dragosh]

Understandable, thanks! Appreciate the work!

[jamespearson04]

ConfusedPolarBear's intro-skipper plugin highlights the real obstacle at present - displaying buttons/text boxes/anything else during playback. The plugin seemed to work, and active development appears to be in progress on one of it's forks, but it requires a custom webui to display skip buttons etc.

The ability for plugins to display buttons etc during playback would make features such as intro/outro skipping implementable purely via plugins, and IMO would be significantly more beneficial than just a native implementation of those features. It would open up the possibility of plugins similar to Amazon Prime's X-Ray for example.

[foux]

I don't think this has anything to do with this conversation. The goal is exactly the contrary : the plugins only creates the marker in an unified way. Clients can then use those generic markers to display the buttons. The display mustn't be made by the plugin, the plugin only records the segments.

[jamespearson04]

It is somewhat tangential I admit, but everything short of the displaying of a skip button is achieveable purely through plugins as is. I don't see any reason that plugins mustn't display things, and it could be a seperate plugin to the tagging plugin. I don't oppose a native implementation, but allowing plugins to handle buttons would achieve the same result with more benefits, vastly empowering the plugin system.

I don't deny it's significantly more complex to implement than just a native button, but I think it would be worth it.

[foux]

This seems to me a slippery slop (to allow plugins to display elements). First it is very hard to achieve on all players who each have very, very different UI, and this might open the gate to a lot of visual discrepency on the apps. In any case it can of course be a discussion, but IMHO it seems to me that this discussion is not the place to have it.

[jamespearson04]

I don't know if I'd describe it as a slippery slope, but you're certainly correct in that it would be a vast undertaking and would require lots of careful consideration for varying platforms. However a lot of the same consideration will have to be taken with regards to native buttons also.

But yes, I suppose you're right. I've just been looking into intro skipping for the past few days and just came across this discussion, and thought it might be helpful to draw attention to the fact that the actual displaying of the button seems to be the last remaining obstacle to intro-skipping via plugin (excluding the use of a custom webui). My opinion on plugin buttons vs native buttons is just a little bonus ;)

[nielsvanvelzen]

We have a strong wish to implement media segments for Jellyfin 10.10. The current state is kinda complicated with this proposal, the somewhat similar pull request and the rebased variant. To make the situation even worse I've made an updated proposal with slightly more detail. Some of the functionality available in the existing pull request is moved to a "possible future extensions" list to avoid complicating the initial change too much, it's highly possible some of those items will still make it to 10.10 though.

Initial implementation (MVP)

The "core" of the media segment implementation. This should be split over multiple pull requests to make it easy to review.

• New concept "media segment"

  • Contains an server-wide unique id
    • The server should be able to figure out which source provided the segment based on this id alone
  • Contains a start and end position (in ticks)
    • The end position must be the same as the start position if there is no duration
  • Contains a pre-defined type
    • Intro, Outro, Recap, Preview, Commercial, Unknown (default)

• Each media source can have 0..n segments

  • Segments are timestamped and always returned in ascending order

• The API returns a new "HasSegments" boolean in MediaSourceInfo

• The API can return segments via a new "MediaSegment" API

  • Always requires an itemId (which is the media source id)
    • No functionality to list everything at once, always item based for item permission checks

• Plugins can provide segments for items via a new SegmentProvider interface

  • Segments are provided during metadata scan, they are part of the item metadata
  • Segments are always cached in the Jellyfin database, it is never retrieved on-demand
  • A segment provider can be enabled or disabled per library, similar to existing metadata providers

Possible future extensions

Should not be part of initial changeset and discussion on these should be postponed to when the MVP is ready.

• Suggest action to take for this segment (e.g. muting or skipping)
• Allow management (create/update/delete) via HTTP API
• Add an annotation segment type
• Add more segment types based on EDL/SponsorBlock definitions
• Read segments from edl files (possibly via plugin)
• Reject provided segments (e.g. they'll be kept in the database but are no longer active)
• Allow push based segment providers. Some plugins may want to provide segments during a scheduled task instead of during metadata retrieval.
• Add a set of default user settings like "AlwaysSkip" (list of segment types to always skip) that clients can use for consistent behavior

Possible official plugins

• Intro detection
  Analyze the media file to find various segment types in a media file, similar to the popular third-party "intro skipper" plugin

• Chapter segments provider
  Use the chapters of a media file to provide segments. The type may be inferred from the chapter names.

[foux]

Here is the list of types I had distilled from edl, sponsorblock, and what was discussed here: https://github.com/thornbill/jellyfin/blob/650447833b6e173933b129444ab1ad002a455bf3/MediaBrowser.Model/Entities/MediaSegmentType.cs

It seems quite extensive. But why the hardcoded mute/skip?

[Shadowghost]

I do agree with most of the modified proposal, but have two comments to make:

• I think we should rather make chapters and segments media source specific than not
• Disabling providers per library should be part of the initial implementation because it might impact scans and disabling it by plugin settings or removing the plugin seems kind of hacky

[thornbill]

Those were to support edl files: https://kodi.wiki/view/Edit_decision_list

[nielsvanvelzen]

I deliberately shrinked the initial list of segment types down so we can get started with some basic types first. We can then have the discussion on which additional types we want and how they'd fit into Jellyfin.
And this could all still happen for 10.10, just trying to reduce the scope of the initial changeset.

I think we should rather make chapters and segments media source specific than not

I've updated the proposal for this. It slightly complicates retrieval as we can't use ItemFields to optionally request them now. I've opted for adding a new "HasSegments" field in MediaSourceInfo. The client must then do an additional request to get the segments from the MediaSegment API.

The MediaSegment API will still use itemId because the id of a media source is the id of an item of type Video/Audio.

Disabling providers per library should be part of the initial implementation because it might impact scans and disabling it by plugin settings or removing the plugin seems kind of hacky

Good point, moved it up to the initial implementation

[she11sh0cked]

Most if not all of the default user settings should probably be "Do Nothing" as to not change previous behavior. Imagine the issues people will create if something suddenly gets muted or skipped.

[JPVenson]

The Current state of the MediaSegments are tracked in the Media Segments Project Board

https://github.com/orgs/jellyfin/projects/63/