Possible memory leak when playing video in DRM

[gustavqmatic]

We use Cog on Intel hardware and DRM as platform.

We see that the memory slowly but steady goes up over time with 1-2mb per minute. We play a list of short full screen video files in a loop.

The trouble shooting we have done so far

We enabled WEBKIT_SAMPLE_MEMORY and sampled over a few hours. We could see that “Resident Set Bytes” and “Total Committed Memory” goes up and “Available Bytes” down but no specific details of what is causing it.

We used the Inspector and Time Interval tool to record memory usage. There we could see that “Page” goes up. What is included in this? Is it possible to get more details on this?

We tried to use different plugins for gstreamer for video playback but found no difference. (Both the new an dold VA-API and software renderer)

The versions we use are:
Cog 0.16.1
WPE WebKit 2.38.5
Built with Yocto Kirkstone

Are there other tools/features we can enable to understand what is eating the memory? Any known issues/patches we could try?

[philn]

Please test WPEWebKit 2.40 and WPEBackend-FDO 1.14.2. To investigate mem usage I'd advise trying with Valgrind's massif, or heaptrack...

[gustavqmatic]

Thank you for the suggestions. I will first try the new versions. Do you plan to release updates in meta-webkit for these versions? I'm trying to bump up the versions myself but I get compile errors.

[gustavqmatic]

We were able to update the Yocto files (some new dependencies was needed) and use the suggested version (inc Cog 0.17.1) but it unfortunately did not help and we still see the memory usage slowly but steady going up.

[philn]

Then I'm afraid the next step would be to try with Massif... Perhaps with the WebInspector you can also find JS leaks. It's very easy to leak in JS... Is this also happening on desktop builds? If you can share the HTML/JS code maybe I can try here

[gustavqmatic]

very_short_video_with_player.zip

Attached is the short html video player example with the clips we play to see the error.

[gustavqmatic]

Here are two heaptrack logs. They are both for WPEWebProcess. I used the attach option in heaptrack.

heaptrack.WPEWebProcess.1649.gz
Download: https://qmatic-my.sharepoint.com/:u:/p/gustav_evertsson/EVS6DBWvIL9DvVODS9WfdwsBdDDuzttW5OS-Ex3_9-ikVQ?e=GdjpPh
Running for ~15h
The memory increased with 730mb according to top in Linux.

heaptrack.WPEWebProcess.2770.gz
Download: https://qmatic-my.sharepoint.com/:u:/p/gustav_evertsson/ER56sVybjeNBhDN-z7UQfXUBhqlQvLhuotM79L8ln_iChQ?e=gzIDpB
Running for ~22h
The memory increased with 1GB according to top in Linux.

To me it looks like gstreamer related, within gstreamer or the interface for gstreamer from webkit. But I know too little about this to understand where the error might me. Do you have any suggestions?

[philn]

Interesting! Would it be possible for you to rebuild WPE with debug symbols? Then heaptrack would provide more details about the leak.

[philn]

Can you also try this? http://sprunge.us/Ef9O96

[gustavqmatic]

I have built with debug symbols but I can not get it to work with heaptrack for some some reason. It does not complain over missing symbol files anymore but still only list them as unresolved functions in the gui. (The symbol files are under /urs/lib/.debug/ and looks ok to me.)

I tested with you patch, unfortunately I can not see any major change. Here are new heaptrack file: https://qmatic-my.sharepoint.com/:u:/p/gustav_evertsson/Edr9CXwmgyJCqr2ZXIRIc0QBUTtkOHfBYpuxa8TDP8UnmQ?e=fUtlnT

This run was for 5h.

[philn]

BTW I can't reproduce this issue in a GTK desktop build of WebKit main. The massif graph is more-or-less flat, but I didn't run this on 12 hours, only a 2 playlist cycles.

If you really need this issue fixed ASAP, perhaps you'd like to contract our consulting services ;)

[gustavqmatic]

I have now been able to get heaptrack with debug symbols. It is just for 45 min but I think it still shows the same issue. Here is a screen shot from heaptrack. As you can see it looks like it is audio related. The audio works and audio from hdmi output sounds correct. We use ALSA for audio handling. Are there any parameters or patches related to this that we can try?

[philn]

Can you expand all the way the configureAudioBin branch?
Anyway, if you want a temporary workaround you can disable webaudio...

[gustavqmatic]

Here is a new screen shot

[gustavqmatic]

I'm pretty sure I disabled webaudio and rebuilt everything but I can still hear the sound from the video playing. What does webaudio do and what is the expected effect of disabling it? How can I verify that it is disabled in the binary?

From configure log
-- Enabled features:
......
-- ENABLE_WEB_AUDIO OFF

[philn]

webaudio is not used by your code, but if you open the code of MediaPlayerPrivateGStreamer you will see that configureAudioBin is wrapped by a ENABLE(WEB_AUDIO) ifdef... That's why, in your specific case, as already mentioned, this can be a workaround to avoid the leak.

What does webaudio

Check the spec: https://www.w3.org/TR/webaudio/

[philn]

Try this? WebKit/WebKit#13965

[gustavqmatic]

That sounds great that you have fixed a leak. We have disabled webaudio as suggested and it seems to work for us. We, at least for now, does not need it so it is okay. We will test the new version, but unfortunately does not have time for that right now.