Massive CPU & RAM Usage - Long Processing Time

[nebutron]

Using the latest AUR developer version (1.0.1.r3.g621) (on Antergos Linux) the program opens and runs just fine but after I am done recording and the encoding/processing occurs my system comes to a halt.

My CPU is at 99% (intel i7 4770K Quad Core) and my RAM which would be 3/16GB is now using 15/16GB. It then takes almost 5 minutes (if not longer) for the recording to actually finish processing.

The video ends up being fine, but I must say it being 15MB is a bit much (not sure if this is related to the issue or not). I know of other software that does the same thing and the same length video would be 1.5MB.

Hoping there are some logs I can find for you to help you figure this out. Let me know!

[phw]

Yes, this is already known and there are a couple of issues about this already. It comes down to imagemagick's cinvert using basically all the resources it can get when optimizing the final GIF. Looking into how limiting the resources for convert affects the processing is on my to-do list.

I would be really curious which software that is you claim produces GIFs only 10% the size for the same recording.

[nebutron]

That is good it's being looked into. Really need to have this working.

As for the size. I mention it in the sense that I know it is possible, but it isn't a software available for Linux so it isn't really comparable (also not open source).

It's called CloudApp (getcloudapp.com). It does a few things, but selecting an area, recording it (either as gif, or video) and then auto uploading with a link. All takes a few seconds.

Now obviously the sizes do vary depending on how long, but I have downloaded quite a few of them to see the size and the largest was 30MB, but the average was 2MB.

Your recording may be higher quality though (but I'd say hard to tell). So I think ultimately what I would like to see would be a quality option. Turn it down kind of thing to be faster or higher quality.

(getting off topic now I suppose)

[phw]

What you can currently do is turn down the framerate. For a lot of recordings you can get away with 7 or 8 fps and still get decent results (it is an animated GIF after all).

But there is definitely room for improvement and this is planned for the 1.1 release. Currently I am just tight on time again and this needs some playing around with different settings, so not sure how long this will take :)

[nebutron]

Does the same issue occur on the non developer version? Heavy system use and all that? As you know, I started another ticket for the regular version about it not launching. If I can get it working, just curious if it has the same problem here.

[phw]

Currently there is not much difference between the development and stable version, so yes, same issues apply. It all comes down to ImageMagick's resource usage here :(

[nebutron]

Thanks. I will keep using this version then.

I hope it can be fixed soon-ish :)

[diwu1989]

When the convert command is called, i think we should be passing in memory limit to be a sane amount, such as 64MB

see documentation on -limit
http://www.imagemagick.org/script/command-line-options.php#limit

peek/src/post-processing/imagemagick-post-processor.vala

Lines 38 to 42 in c0300fd

	"convert",
	"-debug", magick_debug,
	"-set", "delay", delay.to_string (),
	"-layers", "Optimize",
	"-define", "registry:temporary-path=" + temp_dir,

[phw]

@diwu1989 Yes, definitely the resource limits have to be set, see #112 . But definitely not 64MB :)

Evaluating and testing this is on my todo list for the next release

[diwu1989]

Right, as long as it doesn't eat up 4GB on a machine that only has 8GB of ram, thanks

I also wonder if we can tell Imagemagick to be more lazy about optimization, I don't necessarily care as much about the file size, but more about the speed.

Graphicmagick might also be a good alternative to imagemagick for performance speedup

[pohmelie]

This is huge problem, and should be fixed with highest priority. Cause app became unusable. My ubuntu freeze for more than 10 minutes and it looks like it wont stop soon.

[RobertMatkulcik]

Same problem here...

[trelemar]

Stopped using Peek because of this. Sorry devs, my computer freezes if I record a gif > 30 fps and longer than like 8 seconds.

[phw]

I am currently very tight on time, I hope I can work on this soon again. Sorry for the long time without update on this.

Given the comment of @trelemar I just wanted to point out that recording GIFs with 30 fps or more is probably a bad idea, unless you keep it very short or small. You will quickly end up with a file of several hundred megabytes in size, I don't think this is of much use. There is a reason why the default is 10 fps. Personally I would recommend a framerate between 8 and 15 fps.

If what you are recording really requires a framerate that high use a proper video format, Peek supports WebM and MP4.

[xereda]

What strikes me is the performance of the "recordit" app (Mac and win). It manages to generate animated gifs, with HD resolution, and still upload to your server. This all without consuming too much processor and machine memory. Maybe it would not be worth a study on him !?

[dandv]

Same problem with the latest Peek version from the Ubuntu repos, but it got worse. Not only does my 16GB RAM machine freeze, but the animation isn't saved. I've wasted half an hour by now because I kept saving over the same filename, and kept seeing the old screencast.

[jsoftwareengineering]

Another part of this issue is that the user doesn't get any visual feedback about what the state of the program is.

For example, I think I closed peek before it was done saving some larger gifs since I didn't know it was still doing something. Perhaps some kind of animation could play while one should be waiting (like this saving gif which I found using a google image search and so probably couldn't be used).

If you are interested in pursuing this, I could make something that you can use or you could likely find something with a clear and free license.

[dessalines]

Same problem here.

[phw]

Ok, still struggling with this. The main problem is, that I have two issues to fight:

1. This one, where the resource usage eats up all RAM and causes swapping
2. The direct opposite, where default ImageMagick resource restrictions are too strict (convert fails due to resource limits #112)

I have now introduced a change where I actively set the resource limits for disk usage (unlimited, will use whatever is available) and memory usage. For memory usage I use 80% of total available memory, but that might be too much on lower memory systems. Maybe I should use 80% of MemAvailable (at time of calling convert) instead? That would still leave some room for other processes to grow. Any suggestions?

[phw]

After some testing new metric: Convert gets a memory limit of 90% of the available memory at time of invocation. I think this will scale. But I keep this issue open for some feedback. Would be happy if somebody could try the current master version.

Some observation: It is really important to give convert enough memory. Giving it less will just cause it to use more disk space and things get slow quickly.

Another observation: All tools using convert to create GIF files while using the -layers Optimize flag suffer from this. Despite claims that other tools using convert do this faster / more memory effcient (e.g. in #177) I have found no evidence this is the case and was always able to reproduce it with whatever tool I tested.

Bottom line: The GIF file format really isn't made for your HiDPI full-screen 10 minute recording you plan to do. The end result will not be to your satisfaction no matter what I do here. Use WebM or MP4 for this, use GIF for small recordings of screen areas. I should make this clearer in the FAQs.

[phw]

I close this issue now as I think the 1.1.0 release manages the RAM usage well enough. It can't do magic though, ImageMagick just works as it works and the GIF format is in the end just not optimized for this kind of usage anyway.

[orschiro]

Thanks for your hard work, Philipp!

[phw]

I reopen this, as I think I am near a better solution. I experimented with using ffmpeg instead of ImageMagick for conversion, and the results are promising.

If somebody of you could test the latest git revision and run Peek with PEEK_POSTPROCESSOR=ffmpeg set:

PEEK_POSTPROCESSOR=ffmpeg peek

This will force the usage of ffmpeg for generating the final GIF. In my tests it used far less resources, took less time and often generated a smaller file. Also files looked fine, but I haven't tested too many scenarios for now.

If this is working well I could make it the default. Also this would allow some more optimizations, e.g. using compressed temporary video files, saving a lot of temporary disk space.

Note: I am not yet sure how new the ffmpeg version needs to be. Older versions did not do well with generating GIF output. So if you post any results, please let me know the output of ffmpeg -version

[orschiro]

Happy to test! Can I install this version from the ppa or Deb file? El 12/10/2017 12:58, "Philipp Wolfer" <notifications@github.com> escribió:

…

Reopened #125 <#125>. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#125 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKbw6ma4Y3SjDDB3fhHC_3pxgo2DDoLks5srfC_gaJpZM4MsAtG> .

[phw]

@orschiro Thanks in advance :) The daily PPA should work, it looks like it already built the latest code.

[orschiro]

Impressive! Although there seems to be a quality decrease, right?

In any case, performance and stability is great. :-)

[phw]

Impressive! Although there seems to be a quality decrease, right?

Thanks for testing. Not sure about the quality, but that's my main concern and the reason I am asking for feedback :) My quick tests so far looked pretty comparable.

[orschiro]

Let's wait for more people to give feedback then ☺

[luizrogeriocn]

I tested with the daily PPA using PEEK_POSTPROCESSOR=ffmpeg and it worked, although it got a bit laggy during the convert phase.

Before using that, the convert phase would just freeze everything and I had to force reset the system.

[diwu1989]

ffmpeg is way better, please switch to using it for default!
as for quality, we can tune ffmpeg configs later for that, really just care about stability and performance first.

[phw]

Thanks a lot for testing and giving feedback. I will definitely switch to using ffmpeg. My tests so far are pretty good, and if quality issues come up we can experiment with different filters, but the defaults seem to be good already.

I will keep around the ImageMagick post processing for a while so people can use it as a fallback when issues arise, but I will probably drop the ImageMagick code later. As a side effect I will likely also remove the optional libav-tools backend entirely, as this would still require ImageMagick to generate the GIF.

... although it got a bit laggy during the convert phase.

Yes, noticed that too. I will launch the post-processing or maybe the entire recording in a proper thread. So far I came away with calling everything asynchronously (the real work is done in separate process anyway when the external tools get launched), but also forking off external processes adds some notable lag to the UI.

[phw]

The latest code works quite well. Now uses gifski by default, if available (see #212). I won't invest any time adding threading support to the recording process, the current asynchroneous implementation works quite well for me currently. I will revisit this if this becomes necessary.

[xstable]

I'm currently on 1.2.1 and a short gif-screencast (with FPS 10) was freezing my i7 (12GB RAM) for > 5 Minutes (before I hardreset).
From which Version does the ffmpg hack work?

[orschiro]

Hmm... 1.2.1 should already use it, right @phw?

[xstable]

OK, yes, seems to work.
So please make ffmpeg as default, so that there is no need to set the Env-Variable for ffmpeg

[chanoch]

Changing profile.xml to allow a file size limit of 10GiB instead of 1GiB also worked when changing the color profile of an image from RGB to CMYK btw.

[IEWbgfnYDwHRoRRSKtkdyMDUzgdwuBYgDKtDJWd]

For anyone wondering how to fix this automatically, like in a script. I did.

sed -i 's/disk" value="1GiB/disk" value="10GiB/g' /etc/ImageMagick-6/policy.xml