Proposal for Introducing Transcoding Options to Media Downloads

[nathangur]

Current Implementation

The current implementation of media downloads in Jellyfin directly provides the original media files without considering the client's device capabilities or preferences for file size and format. This approach, while straightforward, does not accommodate scenarios where a client might benefit from a transcoded version of the media, either for compatibility or bandwidth reasons. To address this, I propose enhancing the media download functionality to include transcoding support, leveraging existing API endpoints with hopefully minimal to no impact on client compatibility.

Currently, the /Items/{itemId}/Download endpoint, as defined in LibraryController.cs, directly streams the original media file to the client. This is done by the onDownloadClick function in the web client.

Proposed Enhancement

The proposal is to extend the functionality of the existing download endpoint to support optional transcoding based on the client's request. This can be achieved by introducing an optional query parameter or a POST request body that allows clients to specify their transcoding preferences. This would entail an added dropdown or menu for a client to select the download quality per file or as a per user setting.

Option 1: Utilizing Query Parameters

We could extend the existing GET endpoint to accept additional query parameters that dictate the transcoding requirements. For example, parameters could include desired codec, bitrate, resolution, etc that are grouped by a selection of quality settings. This approach maintains the simplicity of a GET request but might limit the complexity and functionality depending on the future needs of this endpoint.

Option 2: Introducing a POST Endpoint

Alternatively, we could introduce a new POST endpoint or repurpose the existing endpoint to accept a POST request with a body containing the transcoding preferences. This approach offers more flexibility but is a more major API change.

Download requests would be handled by a transcode job that would save the files in the transcode file path.

This aims to be the base for having offline downloads and syncing for clients. The lack of support for offline play is also one of the major turning points away from switching to Jellyfin and one of the most requested features.

Related feature requests:

Support Downloading of Transcoded Files
Offline Mode Support on Android
Offline Sync Feature

Any ideas for implementations or feedback would be appreciated :)

[felix920506]

Not familiar with the technical stuff, this is how I understand it:

• Allowing transcoded downloads means the server has to transcode the media for the client to download
• Transcoding can take a while for longer videos and/or on low end devices
• From what I understand, most containers don't like being used without the encoding completing first
• Does this mean the client will have to sit at a waiting page with an open browser window?
• Or alternatively, how would the user be notified when the transcoding is done and ready for download?
• For the latter solution, Storage isn't infinite. How would JF handle requested but never downloaded transcodes to prevent the transcode directory from filling up?

[nathangur]

• Allowing transcoded downloads means the server has to transcode the media for the client to download
• Transcoding can take a while for longer videos and/or on low end devices

Yes this is true, but it's no different from transcoding on the fly for a client to watch. Obviously if you are running your server on a rpi it's not going to do so well.

• From what I understand, most containers don't like being used without the encoding completing first
• Does this mean the client will have to sit at a waiting page with an open browser window?

Again, it shouldn't be much different than transcoding on the fly. I haven't looked into it too much for transcoding and downloading simultaneously, but I reckon it's possible.

• Or alternatively, how would the user be notified when the transcoding is done and ready for download?

This would be another added UI element. If Jellyfin had a global notification system with little bubbles or blurbs on the top right for instance it could be displayed there. If I remember correctly there was some sort of notification system being worked on but I don't know the state of it.

• For the latter solution, Storage isn't infinite. How would JF handle requested but never downloaded transcodes to prevent the transcode directory from filling up?

Jellyfin has a clear transcode directory that runs at a user set time or by default every night (again if I remember correctly. I keep mine disabled) so this isn't an issue.

[nathangur]

Well this is off topic, speaking to the notification system a little bit more. I would suggest an implementation of a system like the one I shared above. This system would share info to users of added media, server maintenance or issues, new updates and features, and other things that are running more per item basis like download status. I don't think this will be possible to be under the 11.0 update.

[JPVenson]

Could you answer me please how this would be any better then just the client requesting a normal transcode and then storing the resulted stream into a container itself instead of displaying it?

[nielsvanvelzen]

We already have an API that returns transcoded streams, which can be used instead. Why add a new endpoint in the already bloated API?

[nathangur]

I'll start with the web client then and proceed to look at offline sync for the android app first.

I'm not sure why any changes would be needed in web. Apps can provide custom download functionality already by adding a function to the native shell (the JavaScript object injected in web by the app). In fact the iOS app already has an offline implementation in development (behind a toggle in unreleased code) that uses the transcode api.

This proposal has a similar use case, which is to batch transcode certain media in advance to lower resolutions or a compatible format and save them on the server by selecting options on the web UI. So it's not just about deleting the transcoded file after downloading it.

I'm not sure exactly what you are referring to but that isn't my proposal. Sure it could be done in the future, but my current focus is on transcoding downloads and offline sync.

[nyanmisaka]

https://features.jellyfin.org/posts/570/pre-transcoding
https://features.jellyfin.org/posts/57/support-download-of-transcoded-files

IMHO downloading the transcoded video function can be split into a pre-transcode request and download.

[nathangur]

I see now. I think pre-transcoding might work best as a plugin though.

[nyanmisaka]

I see a high possibility that two things with overlapping functionality can be done in one PR.

The biggest difference between the two is the path to the transcoding target - the temporary folder and the source file subdirectory with write permissions.

Similar to what you'd expect from the proposed Web changes. It is just one more checkbox - save the transcoded files to the original directory and keep the naming convention.

[nathangur]

To add to the importance of having an endpoint for transcoding downloads, we've made several attempts to use the existing stream endpoints and place the stream in a container to be downloaded. But this leads to issues with playback such as the SeekHead not being in the correct position (at the end of a mkv file) and other player related issues. This isn't an issue if you let the server transcode the file first and then download it, but for large files and low-end hardware this isn't ideal. Bobby proposed a few solutions which ended up needing modifications to the existing endpoint or creating a new one anyways. If transcoded downloads are something that multiple clients need/would benefit from, wouldn't a server change be better than changes to multiple clients with different implementations on/for that OS? Which if the developer of that implementation were to leave then having more code that isn't universally understood is a bigger detriment to the project than another API addition.

That's my two cents anyways :)

[bobby3605]

I've been working with @nathangur on this. I've found that for mkv files, the SeekHead element gets written by ffmpeg after the transcoding is complete, because it needs to know the position of everything in order to write the SeekHead info. So downloading it while it is transcoding gives you an unseekable video on exoplayer. mpv and other players can read the Cues data at the end of the file, so they still work, but exoplayer won't do that, and it should have the SeekHead, because searching for the Cues data can be slow, especially for large files.
https://exoplayer.dev/troubleshooting.html#why-are-some-media-files-not-seekable
https://exoplayer.dev/doc/reference/com/google/android/exoplayer2/extractor/mkv/MatroskaExtractor.html#FLAG_DISABLE_SEEK_FOR_CUES
There is also an issue where the arguments for ffmpeg are disabling the metadata and chapter information. This results in a broken total duration and no chapter information in the transcoded file when playing it with exoplayer. Setting map_metadata and map_chapters to 0, instead of -1, fixes this. I'm not sure why it was set to -1, but it's probably important for something.
https://github.com/jellyfin/jellyfin/blob/master/MediaBrowser.Controller/MediaEncoding/EncodingHelper.cs#L6694

I can think of a few potential solutions to the problem of ffmpeg writing data back.

1. Since ffmpeg reserves a fixed size at the start of every mkv file for the SeekHead, after ffmpeg finishes transcoding and jellyfin has downloaded the file, the first n bytes of the transcoded file in the cache can be redownloaded, where n is the position of the end of the space ffmpeg reserves. I tested this myself using dd to copy the bytes, and the final file played with no issues. This would have to be adjusted for any other file types that ffmpeg has to write back data. Since only 213 bytes or so are needed for mkv files, an api call that returns the raw n bytes could be added, and then the client would override those bytes in the file it downloaded.
   https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/matroskaenc.c#L862
2. Wait until the file finishes transcoding before starting the download. This option will always work for all file types, but you lose the simultaneous transcode and download. To implement this effectively, a new api call that queues a transcoded file will need to be added, and the client will probably need to poll the server to check if the transcode is completed.
3. Modify ffmpeg to support outputting to a tcp socket. So the transcoded file is only stored on the client, and ffmpeg makes any changes it needs on the file on the client directly. This may be a complex solution and some firewalls may block this behavior.

I'd appreciate any feedback or suggestions about this.

[bobby3605]

Adding a menu for pre-transcoding is probably the most straightforward solution.
https://features.jellyfin.org/posts/570/pre-transcoding
In this post, @joshuaboniface stated that this would need to be a plugin, rather than a built in feature, but if I add an option in the GUI to transcode a file, then it should work with all servers, not just ones with a transcoding plugin.
I could add a menu for transcoding that allows you to select video and audio tracks and codecs, along with the final file format and any other needed information. The server would transcode the file and keep track of it as an alternate version. Then the download menu can be updated so that you can select alternate versions, if it doesn't already support it.
I do see one potential flaw, which is if you want to bulk transcode and download a season or series. I'd like to add a menu option to download a whole season or series, instead of clicking on each episode individually, but there's not a straightforward way to pick which version of each episode you want in bulk.

[Chaphasilor]

The pre-transcoding solution also isn't a great fit for music clients (tons of files that are quick to transcode). While Finamp already supports that, we still have the write-back issue that we'd like to get rid of. I'm not sure if some sort of HTTP/2 or HTTP/3 based solution would be possible, where the client requests all files they want to download (including transcoding information), and the server pro-actively sends back the files once they're ready.
The problem with that solution is probably that the connection will get suspended by the OS on mobile devices, since it's not a regular download which could be scheduled by the OS.

[foxinsocksinboxonknox]

If the issue is transcoding to a specific container (mkv) why not transcode to a different container, such as webm, mp4, or ts? Subtitle tracks could be extracted and saved as a different file, though that would require the client player to know how to check for them.

[nathangur]

Could the Jellyfin Server team take a look at this? This is at a standstill currently.

[thornbill]

The team is currently pretty tied up with squashing bugs for the upcoming 10.9.0 release. Sorry for the delay! 💜

[nathangur]

The team is currently pretty tied up with squashing bugs for the upcoming 10.9.0 release. Sorry for the delay! 💜

I don't mean to pry, but could this proposal start to be discussed further? Cheers

[bestad391]

The team is currently pretty tied up with squashing bugs for the upcoming 10.9.0 release. Sorry for the delay! 💜

Pretty new to Jellyfin in general and have been scouring the forums to find the latest updates on this feature. Is this still being looked into by chance? Is there another up-to-date forum for this feature?

[nathangur]

The team is currently pretty tied up with squashing bugs for the upcoming 10.9.0 release. Sorry for the delay! 💜

Pretty new to Jellyfin in general and have been scouring the forums to find the latest updates on this feature. Is this still being looked into by chance? Is there another up-to-date forum for this feature?

This is as new as it gets for a server implementation.

[gnattu]

Unfortunately, such an API is far from ideal on the server because transcoding to arbitrary containers isn't always streaming-compatible. This means you'd have to wait for the server to finish the entire transcoding process before even starting to download the file. Transcoding is often slow, and having an API that waits for minutes or even hours depending on transcoding speed is highly impractical.

A more practical approach would be to introduce an "optimize" option, allowing the server to create an alternate version of the video with a specified transcoding profile that can be directly played on the target device. The server could maintain a queue for such transcoding tasks so users can monitor progress on a web page. Once completed, the version can be downloaded using the current API as-is.

Also, you can already achieve this by downloading the entire HLS stream, without any server modification. HLS is designed for streaming, so you can download the video as quickly as the server transcodes it. Although HLS streams are less common for local video playback, many players do support playing local HLS .m3u8 files. They can also be easily remuxed into more common containers like MP4 or MKV if needed.

[GodTamIt]

I've been working on video streaming in my other work, so thought I might chime in here.

I'd +1 on what @gnattu about HLS. CMAF single-file HLS might be a really nice solution here. Here's why:

• If you transcode into CMAF HLS with ffmpeg with flags like -f hls -hls_segment_type fmp4 -hls_playlist_type event -hls_flags independent_segments+single_file -hls_time 6 -movflags hybrid_fragmented -master_pl_publish_rate 1 -master_pl_name master.m3u8 -hls_segment_filename %v.m4s (the master playlist might not be needed here):
  • You can download the video progressively, HLS-style, as if you were streaming. The server would need to support fulfilling byte-range requests for the file, but that is definitely doable.
  • Since the video is streamable, it means you can also have partial downloads in case the user loses internet connection. You can start watching a partially downloaded video, kind of like how YouTube Downloads work.
  • The resulting single-file .m4s is also a standalone .mp4 file that can play without the manifest. It has the appropriate moov box and each segment is independently playable. This means you can download it after the entire conversion is done without needing to remux. In fact, you can just rename it .mp4 if you want and it'll work.
• CMAF HLS also officially supports packaging the main video codecs: H.264 (AVC), H.265 (HEVC), VP9, and AV1, which should be enough.
• Single-file CMAF HLS has pretty strong support in macOS 10.12+, iOS 10.0+, tvOS 10.0+, ExoPlayer (Android), and HLS.js on web.
• MP4 playback is obviously very widely supported.
• Note: I'm running on the latest ffmpeg v7.1, so I didn't check for older version support.