Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Low priority for preview generation #17966

Closed
PaulLereverend opened this issue Nov 16, 2019 · 11 comments
Closed

Low priority for preview generation #17966

PaulLereverend opened this issue Nov 16, 2019 · 11 comments
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap enhancement feature: previews and thumbnails needs info

Comments

@PaulLereverend
Copy link

Hi !
I would like to share a feature request.

Is your feature request related to a problem? Please describe.

Low hardware like raspberry pi or other ARM based cpu become unresponsive when generating previews. When we upload multiple images in a folder, the server has to generate the previews. This uses all the CPU power so the web interface becomes very slow and unresponsive. It is a bigger issue when you have 2 users on the same server.

Describe the solution you'd like

The idea would be to generate previews with a low priority on the CPU, so the interface remains usable.

@PaulLereverend PaulLereverend added 0. Needs triage Pending check for reproducibility or if it fits our roadmap enhancement labels Nov 16, 2019
@kesselb
Copy link
Contributor

kesselb commented Nov 16, 2019

Hey ;)

Sounds like a duplicate of #1732 to me. From a technical point it's not possible to decrease or increase the priority for a single php request.

@go2sh
Copy link
Contributor

go2sh commented Nov 16, 2019

I'am not 100% sure. But the previews should be generated by the cron script. So you should run it from cron (crontab setting and not per ajax or request) and then control either the nice level or cpulimit and/or place it in a cgroup handling it.

@kesselb
Copy link
Contributor

kesselb commented Nov 16, 2019

@go2sh it depends ;) By default previews are requested via http and generated on demand. Some people are using the previewgenerator app to generate the previews by cron.

Please have a look at https://ownyourbits.com/2019/06/29/understanding-and-improving-nextcloud-previews/. Probably some of the recommendations can improve the situation for you.

@sunjam
Copy link

sunjam commented Nov 16, 2019

See this PR, related to the above link.

@PaulLereverend
Copy link
Author

@kesselb I'm using the previewgenerator app from cron at night and it works pretty well. But when we upload images, the previews are generated immediately (due to on demand) which make the server unresponsive.

I already did the recommendations from ownyourbits which improved the situation but I think it can still be improved.

if I'm not mistaken the php request start a few ffmpeg processes to generate thoses previews. Isn't it possible to decrease the priority of thoses processes ?

@dgtlmoon
Copy link

In my opinion this is the wrong way to go forwards, as a systems engineer I can see that when a process (even if it's "low" priority) is going to attempt to read all the data from the disk file to generate a set of thumbnails.

Most NAS/storage systems that nextcloud run on contain spinning disks, you cannot make a spinning disk any faster, you cannot say 'oh well its on low CPU priority', that disk header is doing stuff, it's busy, it's going to affect the whole system no matter what

you need to look at

  • creating less thumbnails
  • creating smaller thumbnails
  • dont create the huge preview even when you might just want to see that jpeg once a year
  • using existing EXIF metadata thumbnails, that will save you from scanning the whole file (my $200 smartphone now creates photos of 12 megabytes!)

@kesselb
Copy link
Contributor

kesselb commented Nov 16, 2019

if I'm not mistaken the php request start a few ffmpeg processes to generate thoses previews. Isn't it possible to decrease the priority of thoses processes ?

Afaik ffmpeg is only involved for movie thumbnails. If you decrease ffmpeg priority I would expect even more lag because with lower priority generation takes more time and the process will block.

@go2sh
Copy link
Contributor

go2sh commented Nov 16, 2019

Ok, so ultimalty this is a feature request for a preview generation architecture, that can be constrained by cgroups.

@kesselb
Copy link
Contributor

kesselb commented Nov 16, 2019

@go2sh #7855

@kesselb
Copy link
Contributor

kesselb commented Nov 16, 2019

@PaulLereverend okay with you to close this issue? feature: previews and thumbnails we have plenty of issues tagged for preview generation. I'm not sure if we need another one ;)

@dgtlmoon
Copy link

dgtlmoon commented Nov 17, 2019

please close this issue, it only exists as someone's guess as to the symptoms of existing problems

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap enhancement feature: previews and thumbnails needs info
Projects
None yet
Development

No branches or pull requests

5 participants