[Proposal] FFmpeg call handing refactoring and FFmpeg remote integration

[JPVenson]

[The proposal is now out of draft status and can be discussed]

Hey,

I would like to propose 2 things that are tight together. As Tim (@TimGels) and I have extensively on discord discussed, we are aiming to provide a more tightly integrated solution to remotely transcode and run ffmpeg. We are aware that there are existing solutions (rffmpeg) but they are not as user friendly as it could be and due to the generalized way are missing on some comfort features then a directly implemented solution.

My proposal is divided into two parts:

1. Refactor all ffmpeg calls into a structured and object oriented way
2. Build a solution that allows the user to manage and jellyfin to invoke ffmpeg on a remote machines

The 2nd point cannot be realized in the existing environment as discussed later.

1. ffmpeg refactoring

As of today there are XXX instances where the ffmpeg executable is either called directly without capsulation, or through statically typed helper methods.

• Emby.Server.Implementations.LiveTv.EmbyTV.EncodedRecorder
  Calls the ffmpeg executable directly with some very specific arguments to transcode some media source
• MediaBrowser.Controller.MediaEncoding.IMediaEncoder implementation MediaBrowser.MediaEncoding.Encoder.MediaEncoder
  Is an encapsulation for the ffmpeg process, does however expose filesystem references to ffmpeg that are used at the TranscodingJobHelper as well as the AttachmentExtractor and SubtitleEncoder
• Jellyfin.Api.Helpers.TranscodingJobHelper
  Is referenced statically from several api controllers and helpers and therefor cannot be replaced at runtime.

I propose a encapsulation of all direct or indirect calls to ffmpeg via a JellyfinFfmpegService. This service provides a structured way constructing ffmpeg commands similar to FFMpegCore. However this implementation would not directly allow specification of primary output data but only provide a new JellyfinFfmpegStream or similar to access the result. With this abstraction it will be possible to later virtualize local and remote ffmpeg calls. Also by providing this low level of I/O abstraction, most ffmpeg specific code can be retained with minimal adaptation.

There are 2 main cases how jellyfin handles the ffmpeg output:

• a. Save to Storage (examples: Subtitle Encoding, Chapter Generation)
• b. Provide via Api (Hls Streaming)

For a. the JellyfinFfmpegStream will provide a method RealiseToStorageAsync(MediaSourceInfo | FileInfo) to obtain the complete stream on the requested storage location. For b. the stream would allow partial, byte exact transmission either on demand or preemptively by accessing the System.IO.Stream methods the JellyfinFfmpegStream implements. This might need some rework on the HLS controller to be optimal as the current FileStream access buffers might be ineffective in this networked scenario.

The whole goal of this refactoring is to enable an in-process handling of ffmpeg requests to be serialized and executed remotely. Currently this can only be done by replacing the FFMpeg executable which makes this non-trivial for non-technical users as well as missing out on some benefits a tight integration would bring, such as the ability to handle all transcoding in memory on the jellyfin server, as in an on-demand scenario only actively requested data will be requested externally, which in turn would reduce the disk interactions drastically as well as gains in responsiveness as the JF hosting service would be freed from any processing and be focused to only serve the main application.

2. Build a Remote FFMpeg solution

While the road to refactoring the ffmpeg access is relatively straight in terms of scope, the proposal for a JellyfinRemoteFFmpegServer (JRFS) is a bit more unclear to me.
The idea would be to either integrate a system into jellyfin-server directly that allows to manage multiple services or with the help of above refactoring build the system on a plugin basis.

The general inner workings would stay the same and as follows: When a ffmpeg job is requested, the request would be serialized and transmitted to an external server. This server would consist of an aspcore runtime hosting a WebApi 2.0 and would have a jellyfin FFmpeg executable located. The Serialized request would then be processed and provided back to the requesting jellyfin server.

Data transfer of the source media files will be handled in the 1st iteration of JRFS by requiring direct access to the storage medium. This can be achieved by ether mounting a remote location to the corresponding JRFS or providing physical access to the files. In the 2nd iteration the JRFS will also support other means of obtaining the media files such as from jf-server directly.

This poses two important questions. Should the JRFS push the requested output back the the jf-server or should the jf-server pull such data on-demand. Both settings have pro and cons.

Push

Pro:

• All requested data would be available immediately when requested by a client causing only negligible delay for transcoding in some cases

Con:

• The Jf-server network would be more congested while streaming as data comes in from the JRFS and goes out to the clients
• Most advantages due to not storing transcoding files would be negated

Pull

Pro:

• No I/O on the storage of the jf-server
• No unnecessary I/O on the network

Con:

• Expected additional Delay in requesting transcoded chunks of data as a request to the JRFS must be made and be processed
• When a temporary network problem arises the effects will be immediately noticeable leading to lag.

In both cases the JRFS aspcore server will be the interface to access any data that is beeing processed.

The 1st iteration should implement any one or both of the options but for a 2nd i am thinking of a peer-to-peer enablement. This would be exclusively for Transcoding jobs that are requested by any client. The idea is to start a remote process and then instead transferring the data intermediately to the jf-server, to point any requesting client to the corresponding JRFS server. I plan to do this by modifying the corresponding api controller and return a 302 to the correct service. However this has some considerable implications, as the required networking setup for the non-technical Enduser is significant, as all JRFS servers must be reachable by the requesting client. This would also facilitate a great number of modifications in the web-api, more or less requiring that the JRFS is a 1st party supported feature.

The JRFS would be firstly provided as a docker container to be installed and configured via environment variables and/or a configuration file. It would register itself at the configured jellyfin-server instance and from then on be available as a ffmpeg processor. For the 1st iteration it is only planned to allow one JRFS to be registered, later iterations might introduce multiple instances with load balancing.

Questions Open

• Would a refactoring as described for the ffmpeg access have a chance to be accepted
• Should such remote ffmpeg functionality be part of jellyfin or better be a 3rd party integration
• Should the integration be based on Push/Pull/Both/Other
• Is a Peer-To-Peer solution viable or not or only as a "pro-user" feature

[sevenrats]

this is a great idea!