Preload API

[joeyparrish]

We should have an API to allow an application to pre-load a manifest and some of the media segments without involving MediaSource or a <video> element. An application could speculatively pre-load the beginning of several pieces of content while the user is making a decision. That content could then begin playing almost instantly.

This requires some redesign. Currently, new Player() requires an HTMLMediaElement, and the load() pipeline connects from manifest parsing all the way to streaming and MediaSource. These things would need to be decoupled, and a new preload() method would have to be introduced.

Note, though, that if we had it today, this new preload() method would not be usable from a service worker just yet, because our DASH manifest parser depends on DOMParser for XML parsing, and DOMParser is not available in a service worker.

Until manifest parsing can be done from a service worker context, operations involving a manifest must be done from the page context.

[joeyparrish]

API ideas, copied from discussion in #961 (comment)

We will have to change the API so that a video element is not required in the Player constructor. MediaSource would be set up later. Examples:

// current flow
player = new shaka.Player(video);  // attach right away, set up MediaSource
player.load('foo.mpd');

// preload flow
player = new shaka.Player();
player.preload('foo.mpd');  // start loading content
player.attach(video);  // now set up MediaSource and complete the pipeline

// wait to set up MediaSource, without preloading
player = new shaka.Player();
// time passes...
player.attach(video);  // MediaSource setup happens now, when you want it to

The HTMLMediaElement argument to the constructor becomes optional, which gives us API backward compatibility. If a video element is supplied, we will automatically attach in the constructor.

[chrisfillmore]

This is a feature we anticipate wanting to provide to our clients. And more than that, I expect we'd want to preload more than just one piece of content. This is basically to support rapid channel switching behaviour, and the like. There's another wrinkle: I also anticipate we'd want to preload some piece of content and then provide it to a specific Player instance. The reason for this is we may have multiple Players going at once.

If I'm told we need this feature, I think I could work on a PR. But given the above I'd like your input.

[chrisfillmore]

Idea, perhaps total brain fart:

/**
 * Create a Preload, which can be passed to any Player so that it
 * instantly has preloaded content.
 * @static
 * @param {object} params Anything it would need to preload content
 * @returns {shaka.Player.Preload}
 */
Player.createPreload (params) {
  // Do preloading work
}

[bennypowers]

I'd really like to be able to pre load segments or ranges from a single manifest

player = new shaka.Player(video);
// preload with optional ranges param 
player.load('foo.mpd', [
  [0, 1234], // time range in ms
  [2345, 3456],
  [4567, 5678],
  6789, // time index of single segment to pre cache
]);
// party 🎉
player.attach(video)

[chrisfillmore]

Another thought about this: it would be nice if Player#load did not require a video element to be set. It would also be nice if we could configure the player to fetch segments or not. A player with no video element attached would then just fetch manifest updates, and (optionally) build a video buffer.

[joeyparrish]

Hi @chrisfillmore, @bennypowers,

Since #1087, a video element is no longer required in the constructor, but it must be attached by the time load() is called. I do not expect to change this, so that the meaning of load() will continue to be "load all into MediaSource".

For preload, I am thinking along the lines of what Chris suggested in #880 (comment), where we return a token that can be used to complete the load process. The player would be capable of starting multiple preloads in parallel, to support speculation about what the end-user will click on. When the user finally clicks on a thing, you can then choose which preload to continue with. The others would then be invalidated.

Maybe something like this:

let player = new shaka.Player();
let token1 = player.preload('foo1.mpd');
let token2 = player.preload('foo2.mpd');
let token3 = player.preload('foo3.mpd');
// time passes...
player.attach(video);
await token3.load();  // like player.load('foo3.mpd'), but we already fetched segments
// tokens 1, 2, and 3 are all now invalidated.
// buffered data in tokens 1 and 2 has been dropped.
// buffered data in token3 has been transferred to Player and MediaSource.

As for Benny's suggestion in #880 (comment), Shaka Player does not buffer multiple independent ranges in general, so I don't plan to incorporate this into the preload design.

Thanks!

[chrisfillmore]

@joeyparrish that looks pretty good, the only suggestion I have is that it would be preferable if the client could explicitly call something like token.unload(), instead of having Player do the invalidation implicitly. This would enable e.g. the user switching back and forth between two or more channels.

[joeyparrish]

I can see how that might be useful for a TV-like scenario: you could keep the next and previous channel preloaded so the user could "channel surf" with low latency.

But, I wasn't thinking of how this would interact with live streams. I was imagining that we could buffer up to rebufferingGoal and then stop. For live streams, we would need to keep the manifest up-to-date and potentially continue buffering as the live edge moves.

It's not undoable, but I had envisioned something much smaller and simpler. We'll need to give this some careful thought.

What would your expectation be for having several tokens representing live streams? What should they be doing while they wait for a call to load()?

[michellezhuogg]

Hello @OrenMe ,

Thank you for your feedback!

1. That's a great question! Are you thinking about having different configs for different loaders? If so, we can provide configurations to choose codecs and languages for Preload API at the Loader level, same as the configurations at the Player level. If both the configurations in Player and Loader are set, the configs on the Loader level would take privilege.
2. Currently, we don't have the pause/resume mechanism. For VOD content, Preload downloads the init and the first segment, so we don't need to pause the downloading process. For live stream content, Preload downloads of the latest segment, and we can add a config in the future to control that if it causes buffering problems.
3. We will definitely add more events to indicate the preloading status.
4. With one or two segments downloaded for each content, our estimate is that memory would be sufficient. We'll take additional storage options into consideration if we see that as a problem.

Let us know if you have further questions or thoughts! Thank you!

[michellezhuogg]

Also, we would be happy to discuss implementation details if anyone is interested in implementing this and create a PR for it. Thanks!

[girayk]

any idea more less when that feature will availible?

[joeyparrish]

Nobody is working on the implementation at the moment, but contributions are welcome. Let us know if you'd like to take on the implementation, or if you have any questions about the design we published. Thanks!

[girayk]

I have no idea to how to implement. :(
Even I couldn't find any info on how I can prefetch video and read that from the cache without SW.

[joeyparrish]

You probably couldn't find any info on that because it wouldn't quite work like that. Preload would be a Shaka Player feature loading media into memory and keeping it buffered there without connecting to MediaSource SourceBuffers. It wouldn't use Cache or ServiceWorker to fetch or store the media.

This is our design doc: https://github.com/shaka-project/shaka-player/blob/main/docs/design/preload.md

No worries if you don't want to tackle this yourself. We will get there eventually, but the team is prioritizing HLS improvements right now.

[zangue]

Hi everyone,

As it appears that this feature request is not currently being worked on, and, considering it's a pretty valuable and anticipated one, I've conducted some investigations and I'd like to share my ideas (an alternate design proposal) with you on how I think we could go about adding this feature to the shaka player. I would appreciate your feedback.

Proposal

The current design document suggest as a first step to extract a part of the load graph in a loader class but I believe an easier first step would be to enable single manifest preloading with a player instance. I believe this is better approach because it's more "accessible" (less code refactor needed) and it will enable preloading single manifest already which, imo, is a valuable feature the shaka player community can start benefiting from. As next steps, we should make critical components reusable and then introduce the loader class to make multiple manifest preloading in parallel possible. In the following sections I will present the steps in more details.

1. Enable single manifest preloading with a player instance

This step can be broken down into two substeps: (a) enable "detached" initialization of the player and (b) add preload capability.

1.a Enable "detached" player initialization

This means being able to load the player without video element:
With the current load graph we always need to attach the player to a video element first. This is not strictly necessary as all the critical components (manifest parsers, media source engine, DRM engine, text displayers) can be refactor to be initialised without media element (actually it's only the media source engine and the text displayers that need said refactor).
The load path will change from:
Detach -> Attach -> Media Source -> Parser -> Manifest -> DRM -> Load
To:
Detach -> Media Source -> Parser -> Manifest -> DRM -> Attach -> Load

So that the following will be possible:

const player = new shaka.Player();
// Init/load the player in detached mode i.e.
// Detach -> Media Source -> Parser -> Manifest -> DRM
// Note that the first two step can be done in the constructor already.
player.load(manifestUri);
// Starts streaming when a media element gets attached
player.attach(mediaElement);

The player will remain backwards compatible with the current API

const player = new shaka.Player(mediaElement);
// Will init/load the player and start streaming right away
// Detach -> Media Source -> Parser -> Manifest -> DRM -> Attach -> Load
player.load(manifestUri);

To make this possible, we will need to remove the media element dependency for the media source engine construction (MediaSource initialization doesn't require media element) and add an attach(mediaElement: HTMLMediaElement); method to provide a media element later on (at which point the object URL will be created and the media element tied to the previously created media source). Similarly the text displayer will be refactored to be provided with a media element once available;
Furthermore, the load graph for media source playback will be changed as described in the following diagram (note that some nodes are duplicated to simplify the diagram).

I've been experimenting with the above and currently have a working PoC for this substep. I'd be happy to submit a draft PR for feedback.

1.b Add preload capability

Now we can do the following:

const player = new shaka.Player();
player.load(manifestUri)

which will walk the load graph like this:
Detach -> Media Source -> Parser -> Manifest -> DRM
We don't have a media element but at this point we already have a parsed manifest and an instance of the streaming engine which is all we need to start preloading. I can see two ways of approaching this:
a. we add a preload() method to the streaming engine to instruct it to start loading media segment into memory. In this case the streaming engine is aware it's preloading and will need to manage preload buffers accordingly or
b. we make it transparent to the streaming by introducing a layer of abstraction between the streaming engine and the media source engine. Said layer will provide an interface similar to the media source engine to append to and manage buffers and will be responsible of processing and eventually appending the media data to media source engine. A minimal interface would look like this

// Manages fetched media data:
// - Keep them in memory while preloading (MS not attached)
// - Push data to source buffer once a MS engine is provided.
// - Provide interface to manage buffered/staged data
const BufferSink = class {
    // Each buffer sink manage one type of content.
    constructor(contentType: shaka.util.ManifestParserUtils.ContentType) {
        this.buffer_ = ... // Some adequate data structure to store data + append context
        this.mediaSourceEngine_ = null;
    }

    // Before media source engine is provided, keep data in memory
    setMediaSourceEngine(mediaSource: shaka.media.MediaSourceEngine) {
        this.mediaSourceEngine_ = mediaSource;
        // We have the MSE now, process buffer data i.e. pipeline data to MSE
        this.bufferWorkLoop_(); // async, similar to operation queue work in MSE
    }
    // Keep data in memory if not attached to the MS engine else pass down data
    // to the MS engine
    async appendBuffer(/** same args list as MS engine*/) {
        this.buffer_.push(...);
        // ...
    }
}

Then, in the streaming engine we can have a buffer sink per media state so that after fetch the data will be appended by calling mediaState.bufferSink.appendBuffer(...). Obviously there is more to it but I hope I could successfully convey the general idea.
After the above step is completed, preloading manifest will be possible. The feature could put behind a config flag e.g. config.streaming.preload;
Note: The part where we initiate the preload can be added as an separate "Preload" step in the load graph.

3. Make resources as reusable as possible

Now that we can preload single manifest, the next step towards preloading multiple manifest in parallel is to make resources as reusable as possible. Among reusable resources I can identify:

• Media Source Engine
• Manifest parsers: Once a parser is created we can keep it throughout the whole player lifetime. E.g. if the previous load creates e.g. a DASH parser and the next load is a DASH manifest we re-use the already existing parser for it (Though, for this to work, the parsers will need to be made stateless).
• Graph walker implementation (not too sure about this one but should be possible)

The above resources will be owned by the player and can be provided to the loaders (see next section) on demand e.g. via an interface.

4. Introduce loader class

Now we will be ready to add loader class as described in the current design document but with the following suggestions:

• Non reusable resources are created by the loader. Non reusable resources include: the DRM engine, the Streaming engine, the manifest, the graph walker instance, the player configuration. When a loader is passed to player.load(), its resources are transferred to the player so that the player have all required resources and the correct context to start playback.
• Instead of extracting parts of the load graph into the loader class, each loader class creates the whole graph (new walker instance) and walks it until the preload step. Once the loader gets loaded (player.load(loader)), the walker (together with the streaming engine and DRM engine) is transferred and taken over by player which then walks the remaining steps in the graph until load. See diagram below for illustration.

Considering the above a loader class could look like this:

// Pseudo code loader
shaka.Loader = class extends shaka.util.FakeEventTarget {
    constructor(playerInterface, playerConfig, manifestUri, startTime, mimeType) {
        ...
        this.playerInteface_ = playerInterface;
        this.playerConfig_ = playerConfig;
        this.streamingEngine_ = ...;
        this.drmEngine_ = ...;
        this.manifest_ = null;
        // walker for the whole load graph. Will be transferred to the player
        // when it's time to load
        this.walker_ = new shaka.routing.Walker({
            playerInterface.getNodes().detach,
            playerInterface.createEmptyPayload(),
            playerInterface.getWalkerImplementation()
        });
    }
    
    start() {
        // Walks the load graph until preload
        this.walker_.startNewRoute((p) => {
            return {
                node: this.playerInterface.getNodes().preload,
                ...
            };
        });
        
        // When it's time to parse the manifest
        this.manifest_ = await this.playerInterface_.parseManifest(this.manifestUri_, this.mimeType_);
        ...
    }
    
    getWalker() {
        return this.walker_;
    }
    
    getDrmEngine() {
        return this.drmEngine_;
    }
    
    getManifest() {
        ...
    }
    
    ...
};

/**
 * @typedef {{
 *   parseManifest: function(string, string): Promise<shaka.extern.Manifest>,
 *   getWalkerImplementation: function():!shaka.routing.Walker.Implementation,
 *   ...
 * }}
 */
shaka.Loader.PlayerInterface;

And in the player it could be used as follow:

// Pseudo code player
shaka.Player = class extends shaka.util.FakeEventTarget {
    ...
    preload(playerConfig, manifestUri, startTime, mimeType) {
        const playerInterface = {
            ...
        };
        const preloader = new shaka.Loader(playerConfig, manifestUri, startTime, mimeType, playerInterface);
        
        preloader.start();
        
        return preloader;
    }
    
    async load(preloader) {
        // Must certainly do things before...
        ...
        // Transfer resources
        this.config_ = preloader.getConfig();
        // this.applyConfig_(preloader.getConfig())?
        this.drmEngine_ = preloader.getDrmEngine();
        this.streaminEngine_ = preloader.getStreamingEngine();
        this.walker_= preloader.getWalker();
        ...
    
        // Invalide loader
        async preloader.destroy();
        
        if (preload) {
            assert(this.walker_.getCurrentNode() == this.preloadNode_, 'Manifest should have been preloaded already!');
        }
        
        // Proceed with load i.e. new from "Preload" to "Load"
        // Note: if we don't have a video element yet, we will wait for `attach()` to start this route.
        this.walker_.startNewRoute((p) => {
            return {
                node: this.loadNode_,
                ...
            }
        });
        
        // Do more things...
    }
};

5. Preload API

Once all the above steps are done the preload API for loading multiple manifests in parallel should be fully functional and the player can the provide a preload() method with the following signature:

shaka.Player = class extends shaka.util.FakeEventTarget {
    /**
     * @param {string} manifestUri The location of the manifest to preload
     * @param {number=} startTime The time in the presentation timeline we should preloading from
     * @param {string=} mimeType Manifest MIME type
     * @param {shaka.extern.PlayerConfiguration=} playerConfig Specific player config
     * @return {!shaka.Loader}
     * @throws
     */
    preload(manifestUri, startTime, mimeType, playerConfig) {};
}

Example usage:

const player = new shaka.Player();

const loader1 = player.preload('foo1.mpd');
const loader2 = player.preload('foo2.mpd', 10, undefined, {streaming: { bufferingGoal: 15 } });
// Can also be done after load() as well
player.attach(video);
player.load(loader1);

Open questions

How should we deal with player events that occur inside the loader instance during preload?

It ended up being a lengthy document, I appreciate if you took the time to go through all of it. Please don't hesitate provide feedback. Thank you!