LH score seems off when comparing implementations

[antoinebisch]

Provide the steps to reproduce

1. Run LH on http://shop2.qa-tests.prestashop.net/new-products.
2. Run LH on http://shop1.qa-tests.prestashop.net/index.php?controller=new-products

What is the current behavior?

http://shop1.qa-tests.prestashop.net/index.php?controller=new-products consistently gets a better score than http://shop2.qa-tests.prestashop.net/new-products.

http://shop2.qa-tests.prestashop.net/new-products is using image lazy-loading and sometimes gets the mention "Defer offscreen images" (see below)

http://shop1.qa-tests.prestashop.net/index.php?controller=new-products does not get the mention "Defer offscreen images" however the page loads many offscreen images.

Also, see this visual comparison of both pages on WebPageTest which clearly shows how http://shop2.qa-tests.prestashop.net/new-products is more performant than http://shop1.qa-tests.prestashop.net/index.php?controller=new-products however the LH score for the http://shop2 URL is 86 VS 92 for the http://shop1 URL. Similar results appear in PSI: http://shop2 URL scores lower than http://shop1 URL.

[patrickhulce]

Thanks for filing @antoinebisch!

I see three points being made here and responses to each are inline below. Let me know if I missed anything.

http://shop1.qa-tests.prestashop.net/index.php?controller=new-products does not get the mention "Defer offscreen images" however the page loads many offscreen images.

This was fixed in Lighthouse 6.0. The linked reports for 6.0 do not have this problem. (See #10259 )

http://shop1.qa-tests.prestashop.net/index.php?controller=new-products consistently gets a better score than http://shop2.qa-tests.prestashop.net/new-products.

The /new-products URL regressed the LCP metric whereas the improvements were made on metrics were minor and were already quite good to begin with (and therefore see little score improvement to compensate).

LCP depends on the loading time of the large product image. The introduction of new dependencies and greater network contention lengthens the time it takes to download this image.

Before	Regressed

Also, see this visual comparison of both pages on WebPageTest which clearly shows how http://shop2.qa-tests.prestashop.net/new-products is more performant than http://shop1.qa-tests.prestashop.net/index.php?controller=new-products

These are not the same page loads Lighthouse is analyzing. There will be variance that causes some page loads to be visually complete faster than others. The load that Lighthouse analyzed had the largest image visually complete at ~3.1s compared to ~3.4s. Also note that LCP marks the time at which the image is completely done loading, and not the time at which is was visually complete. I agree it's not ideal from the user's perspective, but there's not much more Lighthouse can do about it as it's part of the metric definition and not going to change (#10527 (comment)).

See https://www.webpagetest.org/chrome/inspector-20190809/inspector.html?experiments=true&loadTimelineFromURL=/getTimeline.php?timeline=t:200708_X0_49555a4913b6a1ca98925e27cc6e736c,r:lighthouse,c:0,s:1
and https://www.webpagetest.org/chrome/inspector-20190809/inspector.html?experiments=true&loadTimelineFromURL=/getTimeline.php?timeline=t:200708_J7_a6b38db4ee68a57d4c5acf663b117184,r:lighthouse,c:0,s:1 for traces of the loads that Lighthouse analyzed.

[antoinebisch]

Thank you for this fast and detailed answer @patrickhulce. Makes sense!

One thing I'd like to clarify:

http://shop1.qa-tests.prestashop.net/index.php?controller=new-products does not get the mention "Defer offscreen images" however the page loads many offscreen images.
This was fixed in Lighthouse 6.0. The linked reports for 6.0 do not have this problem. (See #10259 )

The issue you linked is the opposite of what I meant. In the http://shop1.qa-tests.prestashop.net/index.php?controller=new-products page, many images are loaded which do not appear within the initial viewport. Even with LH V6, "Defer offscreen images" are not reported:

This is from PSI which is using LH V6.0.0

This is from The WPT LH report using V6.1.1

Is this because somehow on that page LH doesn't think deferring the below the fold images is as important than the other listed opportunities?

Thank you.

[patrickhulce]

The issue you linked is the opposite of what I meant. In the http://shop1.qa-tests.prestashop.net/index.php?controller=new-products page, many images are loaded which do not appear within the initial viewport. Even with LH V6, "Defer offscreen images" are not reported:

Ah, sorry I misunderstood! The linked issue is actually still the answer to this question just not the one you were hoping for 😆

In matching the behavior of Chrome for offscreen images, the audit became much more forgiving and will ignore offscreen images that are still within a few viewports. Only waaaaay offscreen images will be surfaced now.

[antoinebisch]

Ok I see! Great, thank you.