Magento 2.1.3 out of stock associated products to configurable are not full page cache cleaned

[daniel-ifrim]

In Magento 2.1.3 strictly:
When a associated product goes out of stock the product page of the configurable product is not updated. The cache (full page cache type) is not cleaned, especially when indexer mode is set to 'Update by Schedule'/MView is enabled.
This issue is introduced into Magento 2.1.3 because Magento\CatalogInventory\Model\Indexer\Stock\AbstractAction::_reindexRows
sends to 'CacheContext' the product ids of the simple associated product.
In Magento 2.1.2 the blocks of configurable product page had identities (cache identities, function getIdentities()) containing ids of simple associated products too. But in Magento 2.1.3 these ids are no longer included in these identities. Only the parent configurable product ids are in identities.
On short, the configurable product page is not cache cleaned. Same behavior for categories pages that this product is assigned to.

Preconditions

1. Use Magento CE 2.1.3 with or without sample. I used with sample.
   Set indexer mode to 'Update by Schedule' / Mview enabled
   Have all cache types enabled in admin (including full page cache). I used in this test 'built-in full page cache'. The issue should happen when using Varnish too.

Steps to reproduce

1. Set quantity 1 to a simple product that is associated to a configurable product in admin.
   I use a simple product associated of configurable product SKU 'WT09' in Magento 2.1.3 with sample.
   This configurable product has swatches but the issue is the same to configurable products without swatches.
   With swatches you can actually visually see when an option is not available (product is out of stock).
2. Clear cache and reindex all from console to make a clean test
3. Login as customer and add to cart this associated product from configurable product page
   Please note that by doing this you also load the configurable product page to full page cache.
   Loading before checkout the configurable product page into full page cache type is important to demonstrate this issue.
4. Complete checkout
5. Come back after (1-3 minutes) on configurable product page and refresh the page
   Why after 1-3 minutes: to give time to Magento cron to run the cron task:

<job name="indexer_update_all_views" instance="Magento\Indexer\Cron\UpdateMview" method="execute">
    <schedule>* * * * *</schedule>
</job>

Expected result

1. On short, the simple associated product is not marked as out of stock on configurable product page.

The configurable product page should have it's options updated visually. More exactly, the associated simple product unique combinations of options should be marked as 'out of stock'. In the case of sku 'WT09', we should see the 'disabled' swatch for that associated simple product.
If you clean cache from console you'll see the expected result. Cleaning cache from console after each purchase is not a solution.

Actual result

1. The configurable product page is not cache cleaned.
   In Magento\CatalogInventory\Model\Indexer\Stock\AbstractAction::_reindexRows there is:

$this->cacheContext->registerEntities(Product::CACHE_TAG, $productIds);
$this->eventManager->dispatch('clean_cache_by_tags', ['object' => $this->cacheContext]);

Magento\PageCache\Observer\FlushCacheByTags never gets a configurable product id (in the case - checkout with a product).

The categories pages are not cache cleaned too. Same thing, categories pages blocks contain in identities only configurable products ids (no associated product ids).

A possible solution:

In Magento\CatalogInventory\Model\Indexer\Stock\AbstractAction::_reindexRows add to $productIds the configurable product ids just before the line:
$this->cacheContext->registerEntities(Product::CACHE_TAG, $productIds);
Actually there is the variable '$processIds' that already contain the parents ids beside initial $productIds.

What happens specifically with Mview enabled, see bellow.
When a product is purchased and goes out of stock after checkout, in cataloginventory_stock_cl there are 2 rows inserted because of the quantity subtraction that happens here:
\Magento\CatalogInventory\Observer\SubtractQuoteInventoryObserver and \Magento\CatalogInventory\Observer\ReindexQuoteInventoryObserver
Those 2 rows contain only the simple associated product id. That's fine, only that 'CacheContext' should be 'forced' to get the configurable product id too.
I haven't tested yet but with 'Update on Save'/ Mview disabled, $productIds in Magento\CatalogInventory\Model\Indexer\Stock\AbstractAction::_reindexRows contain the configurable product ids because there is in \Magento\CatalogInventory\Observer\ReindexQuoteInventoryObserver:

$productIds = [];
foreach ($quote->getAllItems() as $item) {
    $productIds[$item->getProductId()] = $item->getProductId();
    $children = $item->getChildrenItems();
    if ($children) {
        foreach ($children as $childItem) {
            $productIds[$childItem->getProductId()] = $childItem->getProductId();
        }
    }
}

Without testing when Mview is disabled, I would say with inder mode 'Update on save', the issue is not there.

[daniel-ifrim]

Even if the configurable product id is sent to be cleaned from cache type full_page, there are additional things to do. 'block_html' and 'collections' cache types must be cleaned too by the cache tag of the configurable product's id. The id of the configurable product is added in many cached blocks tags/identities.

[xmav]

@daniel-ifrim Thanks for reporting this issue.
We've created internal ticket MAGETWO-64282 to address this issue.

[eldonbite]

Hi @xmav. Are there already commits for this issue from which we could create a patch? It's just sad that after the issue with large Varnish cache tags (#7657) has been resolved, it seems this issue got introduced as a consequence.

[PascalBrouwers]

It seems we have this issue because we are using Amasty Stock Alert module, I'm not certain.

[paales]

Isn't this issue fixed on 2.1.4? Related issue for bundle products #10236.

[paales]

Disregard my last comment, the issue isn't fixed if the Stock\Item of a product is updated.

I think Magento should consider implementing an IdentityInterface for stock_item as well and that Magento will automatically handle add the identities of all the ExtensionAttributes as well (stock_item is an extension attribute on a product).

Workaround:

<type name="Magento\CatalogInventory\Model\Indexer\Stock">
    <plugin sortOrder="1" name="hoKingdoStock" type="Ho\Kingdo\Plugin\ProductCacheTagStrategy"/>
</type>

<?php
/**
 * Copyright © 2017 H&O E-commerce specialisten B.V. (http://www.h-o.nl/)
 * See LICENSE.txt for license details.
 */
namespace Ho\Kingdo\Plugin;

use Magento\Catalog\Api\Data\ProductInterface;
use Magento\Framework\EntityManager\MetadataPool;
use Magento\Framework\Indexer\CacheContext;

class ProductCacheTagStrategy
{

    /**
     * @var \Magento\Bundle\Model\ResourceModel\Selection
     */
    private $selection;

    /**
     * @var MetadataPool
     */
    private $metadataPool;

    /**
     * @var CacheContext
     */
    private $cacheContext;

    /**
     * @var \Magento\ConfigurableProduct\Model\ResourceModel\Product\Type\Configurable
     */
    private $configurableType;

    /**
     * BundleProductCacheTagStrategy constructor.
     *
     * @param \Magento\Bundle\Model\ResourceModel\Selection                              $selection
     * @param CacheContext                                                               $cacheContext
     * @param \Magento\ConfigurableProduct\Model\ResourceModel\Product\Type\Configurable $configurableType
     * @param MetadataPool                                                               $metadataPool
     */
    public function __construct(
        \Magento\Bundle\Model\ResourceModel\Selection $selection,
        \Magento\Framework\Indexer\CacheContext $cacheContext,
        \Magento\ConfigurableProduct\Model\ResourceModel\Product\Type\Configurable $configurableType,
        MetadataPool $metadataPool
    ) {
        $this->selection = $selection;
        $this->metadataPool = $metadataPool;
        $this->cacheContext = $cacheContext;
        $this->configurableType = $configurableType;
    }

    /**
     * Retrieve array of related bundle product ids by selection product id(s)
     *
     * @see \Magento\Bundle\Model\ResourceModel\Selection::getParentIdsByChild(), that method seems to be broken.
     * @param int|array $childId
     * @return array
     */
    public function getBundleParentIdsByChild($childId)
    {
        $connection = $this->selection->getConnection();
        $metadata = $this->metadataPool->getMetadata(ProductInterface::class);
        $select = $connection->select()->distinct(
            true
        )->from(
            $this->selection->getMainTable(),
            ''
        )->join(
            ['e' => $this->metadataPool->getMetadata(ProductInterface::class)->getEntityTable()],
            'e.' . $metadata->getLinkField() . ' = ' .  $this->selection->getMainTable() . '.parent_product_id',
            ['e.entity_id as parent_product_id']
        )->where(
            $this->selection->getMainTable().'.product_id IN(?)',
            $childId
        );

        return $connection->fetchCol($select);
    }

    /**
     * @param $ids
     * @return \string[]
     */
    private function getConfigurableParentIdsByChild($ids)
    {
        return $this->configurableType->getParentIdsByChild($ids);
    }

    /**
     * Also Register the parent of the stock item after it has been reindexed.
     *
     * @param \Magento\CatalogInventory\Model\Indexer\Stock $subject
     * @param                                               $ids
     */
    public function beforeExecute(\Magento\CatalogInventory\Model\Indexer\Stock $subject, $ids)
    {
        $parentIds = array_merge($this->getBundleParentIdsByChild($ids), $this->getConfigurableParentIdsByChild($ids));
        $this->cacheContext->registerEntities(\Magento\Catalog\Model\Product::CACHE_TAG, $parentIds);
    }
}

[ishakhsuvarov]

Hi @daniel-ifrim
We are closing this ticket now as the fix is already delivered with the linked Pull Request
Thank you