More speed problems thumbnails from S3

[jplehmann]

It's taking 1+ seconds per thumbnail for pages to load on their first hit, when image processing is occurring.

My problems sound very similar to those discussed in #252, with a few differences. I am pulling image sources directly from other sites, and using S3 to store and serve the processed thumbnails (that is, I am not using 3rd party key-value store, but the built-in DB solution). I might suspect this as a problem but had no performance issues of this magnitude before when the only difference was that I was serving the processed images locally rather than from the web server.

Once I moved my MEDIA_URL to S3, using boto 2.34.0 and django-storages 1.1.8, it began taking about 1 second per thumbnail to load each page for the first time. Subsequent load times still seem a bit laggy but are many times faster than this. This is all with django 1.7. It doesn't seem possible for it to be this slow unless it is doing the work 100% in serial.

I have been using sorl-thumbnail==12.1c, but I also tried the version mentioned there as well as the latest 48d34d8. I opened this issue since the one in #252 was supposedly resolved by this point.

It wouldn't surprise me if I'm doing something wrong. Thanks for the help.

[jplehmann]

These were times I was experiencing when running my app locally. When I deploy to Heroku times improve dropping from about 15s to 8s (for 12 thumbnails). But there still seems to be something seriously wrong.

[jplehmann]

Anyone?

[relekang]

How long does it take if you try with local images? How large are the images you pull from other sites?

[jplehmann]

@relekang thanks for responding. I think I have figured this out.

I believe the slow initial hits are due to my using URL image sources, where each image must be downloaded from other sites and then the result uploaded to s3. I'm guessing that the write performance was the new component introduced, compounded by the fact that this processing is done totally in serial as the template is rendered. When I ran without s3, it at least did not have to save results remotely.