Memory leaks during playback of HEVC / H.265 / HEV1 / HVC1 HLS stream packaged into MPEG-TS (.ts files)

[izogfif]

What version of Hls.js are you using?

master / 1.6.0

What browser (including version) are you using?

Google Chrome 128.0.6613.120 (Official Build) (32-bit) from PortableApps

What OS (including version) are you using?

Microsoft Windows 11 Version 22H2 (OS Build 22621.3155)

Test stream

No response

Configuration

{
  "debug": true,
  "enableWorker": false,
  "lowLatencyMode": false,
  "backBufferLength": 5,
  "maxBufferLength": 6,
  "maxMaxBufferLength": 6,
  "frontBufferFlushThreshold": 6,
  "maxBufferSize": 0,
  "liveBackBufferLength": 5
}

Additional player setup steps

Here are notes that I couldn't put anywhere else.

• I couldn't find an open issue by searching for "HEVC" "H.265" and "h265"
• Since HEVC in MPEG-TS is not supported in current release version, I couldn't reproduce the issue on https://hlsjs.video-dev.org/page, but I still had to check the checkbox to post this issue.
• Below is the additional data that might help investigating the error's origins.

Try this:

• Open video in Google Chrome (e.g. in demo page).
• Start video playback and let the video be played for at least 1 minute to produce noticeable memory footprint.
• Open Web Developer Tool in Google Chrome.
• Go to "Memory" tab.
• Select "Allocation instrumentation on timeline" item / radio button.
• Select "Record stack traces of allocations (extra performance overhead).
• Select "localhost:8080: Main" under "Select JavaScript VM instance".
• Click "Start" button underneath.
• Wait for a few seconds.
• Click "Stop recording heap profile" button / icon in the upper-left corner of "DevTools" (for RTL layout the location of the button / icon may vary).
• In the list that appears click "Retained Size" column's header to sort items by size in descending order (larger numbers are displayed at the top).
• In the "Constructor" column find "Transmuxer" item. Expand it until you see this kind of structure:

Transmuxer
  Transmuxer
    demuxer
      _videoTrack
        pps

• Take a note of the number in "Retained Size" column: it's large for all items mentioned above.

Investigations show that TSDemuxer instance used for video track (de)muxing contains large amount of references to (slices of) fragment / segment / chunks of UInt8Array, presumably with video data in its _videoTrack.pps field / member.

For example, if window.hls represents reference to HLS instance, then window.hls.streamController.transmuxer.transmuxer.demuxer._videoTrack.pps is the thing that keeps references to a huge number of UInt8Array instances and doesn't let browser destroy / GC them.

The length of pps steadily grows during the entire duration of playback. For Sintel (see below) the size of this array reaches 878 elements. For longer videos (e.g. around 1 hour with the same bitrate) it can reach as large as 7802 elements before browser tab crashes with "Out of memory" error.

It should be noted that the size of the used portions of UInt8Array whose references are stored in pps is pretty small:
window.hls.streamController.transmuxer.transmuxer.demuxer._videoTrack.pps.map(x => x.byteLength).reduce((a, b) => a + b, 0) says that combined size of all used portions of all arrays is just 5268 bytes (like, just 6 bytes per item).

But the total size of all underlying buffers which these UInt8Arrays take data from is huge:
window.hls.streamController.transmuxer.transmuxer.demuxer._videoTrack.pps.map(x => x.buffer.byteLength).reduce((a, b) => a + b, 0) is whopping 799,707,013 bytes (almost 800 megabytes) - around 900 kilobytes per item.

Notes

• The issue doesn't happen with AVC / H.264 streams of any bitrate and duration. Only with HEVC / H.265 / HVC1 / HEV1 streams. I didn't check whether it happens with fMP4 streams. Even though documentation says that HLS streams that use HEVC / H.265 / HVC1 / HEV1 must use fMP4 format for fragments / segments / chunks, HLS.js still works for MPEG-TS (.ts file extension) streams that use the same codec.
• In case one would like to suggest a workarounds like "re-encode video to AVC / H.264" or "use lower bitrate (and pray that video isn't long enough to cause the eventual crash)": it's out of the question. Support for HLS streams (video on demand-type) of 4 GB+ videos encoded with HEVC / H.265 / HVC1 / HEV1 with MPEG-TS packaging is requested.
• Destroying and re-creating HLS.js instance every 10 minutes of playback to fight "Out of memory" issues would introduce unnecessary lags and inconveniences for the end-user.

Checklist

• The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
• The issue occurs in the stable client (latest release) on https://hlsjs.video-dev.org/demo and not just on my page
• The issue occurs in the latest client (main branch) on https://hlsjs-dev.video-dev.org/demo and not just on my page
• The stream has correct Access-Control-Allow-Origin headers (CORS)
• There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

• Clone / checkout HLS.js repository's main branch.
• cd into cloned repository's folder.
• Run npm install
• Run npm run serve
• Go to http://localhost:8080/demo/ page in your browser.
• Enter player configuration in the right-side text field:

{
  "debug": true,
  "enableWorker": false,
  "lowLatencyMode": false,
  "backBufferLength": 5,
  "maxBufferLength": 6,
  "maxMaxBufferLength": 6,
  "frontBufferFlushThreshold": 6,
  "maxBufferSize": 0,
  "liveBackBufferLength": 5
}

• Click "Apply" button.
• Enter URL for HLS stream that contains MPEG-TS fragments (files with .ts extension) with content encoded using H.265 / HEVC / HEV1 / HVC1 codec and press Enter.
• Let the video play until it finishes. On 32-bit Google Chrome browser issues happen after around 700 megabytes - 1.5 gigabytes of content have been played.

Obtaining sample files for sample HLS stream

Option one: from repository

Checkout the repository containing sample stream files (at the moment of writing it's still being uploaded).

Option two: create stream manually

• Obtain UHD version of Sintel movie (or any other video for that matter): you can download it from official website (here is the direct link).
• Obtain latest build of ffmpeg
• Create HLS MPEG-TS stream that uses H.265 / HEVC / HEV1 / HVC1 codec using this command:

ffmpeg -y '-i' Sintel.2010.4k.mkv '-map' 'v:0' -c:v libx265 -preset ultrafast -b:v 40597530 -g 24 '-f' 'hls' '-hls_time' '2' '-hls_playlist_type' 'vod' '-hls_segment_type' 'mpegts' '-hls_list_size' '0' '-hls_segment_filename' 'seg_%d.ts' '-start_number' '0' '-master_pl_name' 'master-video.m3u8' '-muxdelay' '0' '-muxpreload' '0' '-output_ts_offset' '0.000000' '-segment_format_options' 'movflags=frag_keyframe+empty_moov+default_base_moof' 'manifest.m3u8'

Key frames are placed every 24 frames (i.e. each second). Fragment length is 2 seconds. Only video stream is selected (no audio) because video is the one that's causing memory issues. Bitrate is chosen high to match the bitrate of original video and also because it produces more noticeable results during test for memory leaks (i.e. errors happen sooner, browser crashes faster).

Expected behaviour

• Video plays back until the end (for long videos for at least 1 hour, but better until the end) without stalling (or, if that's impossible, then with infrequent stalling caused by browser/ HLS.js buffering / fragment eviction). It should be noted that network speed / connection reliability / server load / content delivery is not the issue.
• Browser tab never crashes with "Out of memory" / "Access violation" or other kinds of errors.

What actually happened?

One of the following issues occurs:

• Video plays until around the middle and then stalls indefinitely.
• Browser tab crashes with "Out of memory" message.
• When hovering mouse over the tab's title, a pop-up message is displayed saying "High memory usage: 1.3 GB".

Console output

Too long to send here. Will attach later.

Chrome media internals output

Too long to send here. Will attach later.

[izogfif]

chrome-internals-logs.txt
hls-js-console-output.txt

[izogfif]

Apparently, editing src/demux/video/hevc-video-parser.ts alleviates the problem: replace

            if (this.initVPS !== null || track.pps.length === 0) {
              track.pps.push(unit.data);
            }

with

            if (this.initVPS !== null || track.pps.length === 0) {
              const arrayCopy = new Uint8Array(unit.data.length);
              arrayCopy.set(unit.data);
              track.pps.push(arrayCopy);
            }

track.pps still grows over time, but its elements take up a lot less memory.

[robwalch]

@devoldemar Can you take a look at this issue?

For AVC we just use the last PPS

hls.js/src/demux/video/avc-video-parser.ts

Lines 145 to 151 in 45696be

	// PPS
	case 8:
	push = true;
	track.pps = [unit.data];
	break;

This issue may be related to the SPS handling in that it is responsible for resetting the SPS and PPS data but will not do so if it matches earlier data:

hls.js/src/demux/video/hevc-video-parser.ts

Lines 133 to 146 in 45696be

	// SPS
	case 33:
	push = true;
	spsfound = true;
	if (typeof track.params === 'object') {
	if (
	track.vps !== undefined &&
	track.vps[0] !== this.initVPS &&
	track.sps !== undefined &&
	!this.matchSPS(track.sps[0], unit.data)
	) {
	this.initVPS = track.vps[0];
	track.sps = track.pps = undefined;
	}

Regardless, that does not stop more pps data from getting added:

hls.js/src/demux/video/hevc-video-parser.ts

Lines 158 to 159 in 45696be

	if (track.vps !== undefined && track.vps[0] === this.initVPS) {
	track.sps.push(unit.data);

hls.js/src/demux/video/hevc-video-parser.ts

Lines 183 to 184 in 45696be

	if (this.initVPS !== null || track.pps.length === 0) {
	track.pps.push(unit.data);

If all TS segments contain the same SPS and PSS won't this code keep pushing the same data to to those arrays leading to the described leak?

[devoldemar]

@izogfif @robwalch Sorry for the delay. Unlike AVC, in HEVC parser we definitely have to handle a case with several SPS and PPS in the decoder configuration record. If VPS+SPS+PPS are retransmitted, but not used to re-generate initial segment, SPS and PPS arrays should be left intact. In case of SPS I added the following condition (new VPS must be used as initVPS):
if (track.vps !== undefined && track.vps[0] === this.initVPS) {
For PPS the condition should be the same, now I see there's really an omission.