Failure to allocate memory error when importing customers in admin

[piotrekkaminski]

From @kevin-carroll-cl on March 10, 2017 19:51

When importing customers into a site with a large number of existing customers through the standard Magento importer tool (System > Data Transfer > Import), with Entity Type: Customers Main File, and Import Behavior: Add/Update Complex Data, the import POST returns a 500 status caused by exceeding allowed memory limits.

Preconditions

1. Magento Enterprise 2.1.4
2. Server is running PHP version 7.0.15
3. Memory limit is set to the standard 768 MB
4. The target store already has a large number of customers (eg 250k+, we experience it at 261,659)

Steps to reproduce

1. In admin, go to SYSTEM > Import
2. Set Entity Type to: Customers Main File
3. Set Import Behavior to: Add/Update Complex Data
4. Select a valid import CSV with at least 1 customer row, and select Check Data
5. After data validates, select Import

Expected result

1. Customer data should be imported

Actual result

1. Import Screen loader never expires & Import POST responds with status code 500:
   
2. Server error log shows a corresponding PHP Fatal error:

PHP Fatal error: Allowed memory size of 805306368 bytes exhausted (tried to allocate 20480 bytes) in /server/sites/(SITE)/vendor/magento/framework/Model/ResourceModel/Db/VersionControl/Snapshot.php on line 47, referer: https://(SITE)/backend/admin/import/
ImportError.txt

The memory use appears to be mainly triggered by framework level code (\Magento\Framework\Model\ResourceModel\Db\VersionControl\Snapshot::registerSnapshot), and is likely platform independent.

Through profiling and debugging we have found 2 very expensive processes that seem to be contributing to this. When beginning the customer import, \Magento\CustomerImportExport\Model\ResourceModel\Import\Customer\Storage::getCustomerId is called, which triggers \Magento\CustomerImportExport\Model\ResourceModel\Import\Customer\Storage::load to execute which in turn executes \Magento\ImportExport\Model\ResourceModel\CollectionByPagesIterator::iterate. Through this process every existing customer account is retrieved and loaded, which is a costly procedure to begin with, but then the email, website id and customer ids are loaded into a multidimensional array. Additionally, loading the customer array as part of the CollectByPagesIterator::iterate method, triggers snapshotting every customer and loading the whole entity data into another in memory array.

Every existing customer is loaded and initialized as a full customer entity object, and data is held in multiple in-memory locations, in order to import a relatively small collection of new customers. This is an unsustainable practice, is costly in time, processing, and memory, and it is difficult to see the value in doing this.

Copied from original issue: magento/magento2#8851

[piotrekkaminski]

From @sopedro on April 8, 2017 17:28

Hi, same issue here in CE 2.1.5.
I imported a lot of products with no problems and i tried to import customers and i just can import 3 customers at a time.

This is weird, no log results at all, I don't know how to fix it...

Thanks,

Pedro

[piotrekkaminski]

From @IlnitskiyArtem on June 30, 2017 10:6

@kevin-carroll-cl , Thanks for the feedback.
Unfortunately, I could not reproduce the issue in 2.1.5 , 2.1.7, 2.1-develop, 2.2.0(develop) with file in attachment (change file extensions from '.txt' to '.csv').
May please let me know if this issue relevant for latest release of Magento 2? And if you can, attach your import file.
CustomersMainFile.txt

[piotrekkaminski]

From @adriansavuosf on July 19, 2017 7:7

Hi,

We have the same problem as described in the ticket by kevin-carroll-cl . In order to reproduce it you need to have a store that already have inside a lot of customer (our import doesn't work after 100k users and memory limitation of 256 MB). When you are trying to import new customers after a specific number of customers are already imported, you will not be able to import even 1 more customer.

[piotrekkaminski]

From @IlnitskiyArtem on July 24, 2017 14:22

@kevin-carroll-cl Internal ticket MAGETWO-70964, which tracks this GitHub issue, is in our issue backlog.

[piotrekkaminski]

From @magento-engcom-team on September 22, 2017 16:24

@kevin-carroll-cl, thank you for your report.
We've created internal ticket(s) MAGETWO-70964 to track progress on the issue.

[piotrekkaminski]

From @MattDelac on October 27, 2017 9:18

Hi @sopedro

Did you find any work around ?

Thanks
Matt

[piotrekkaminski]

From @MattDelac on October 27, 2017 10:51

As a similar problem, if I import 100 customers, it works but it fails with more than 150 clients (for a total database of 2K existing clients)

An there is NO information concerning the failure

Does someone have any idea of how can I resolve this problem ?

Thank you in advances
Matt

[dmanners]

So I am going to have a look at this. Have looked at a set of 10000 customers and the import is working fine with a memory limit of 268M. I will try to generate a set of 50000 customers and then see how the import behaves.

[dmanners]

Merged under the PR #97