Fix ImportExport bug which occurs while importing multiple rows per entity

[tim-bezhashvyly]

This pull request solves couple of bugs.

As you know Magento allows import of a complex data by splitting the entity information into multiple lines. So let's say I want to import 2 products like this:

sku,_store,_attribute_set,_type,_category,_root_category,_product_websites,activation_information,color,computer_manufacturers,contrast_ratio,cost,country_of_manufacture,country_orgin,cpu_speed,created_at,custom_design,custom_design_from,custom_design_to,custom_layout_update,description,dimension,finish,gallery,gender,gift_message_available,harddrive_speed,hardrive,has_options,image,image_label,in_depth,manufacturer,max_resolution,media_gallery,megapixels,memory,meta_description,meta_keyword,meta_title,minimal_price,model,msrp,msrp_display_actual_price_type,msrp_enabled,name,news_from_date,news_to_date,options_container,page_layout,price,processor,ram_size,required_options,response_time,room,screensize,shape,shirt_size,shoe_size,shoe_type,short_description,small_image,small_image_label,special_from_date,special_price,special_to_date,status,tax_class_id,thumbnail,thumbnail_label,updated_at,url_key,url_path,visibility,weight,qty,min_qty,use_config_min_qty,is_qty_decimal,backorders,use_config_backorders,min_sale_qty,use_config_min_sale_qty,max_sale_qty,use_config_max_sale_qty,is_in_stock,notify_stock_qty,use_config_notify_stock_qty,manage_stock,use_config_manage_stock,stock_status_changed_auto,use_config_qty_increments,qty_increments,use_config_enable_qty_inc,enable_qty_increments,is_decimal_divided,_links_related_sku,_links_related_position,_links_crosssell_sku,_links_crosssell_position,_links_upsell_sku,_links_upsell_position,_associated_sku,_associated_default_qty,_associated_position,_tier_price_website,_tier_price_customer_group,_tier_price_qty,_tier_price_price,_group_price_website,_group_price_customer_group,_group_price_price,_media_attribute_id,_media_image,_media_lable,_media_position,_media_is_disabled,_custom_option_store,_custom_option_type,_custom_option_title,_custom_option_is_required,_custom_option_price,_custom_option_sku,_custom_option_max_characters,_custom_option_sort_order,_custom_option_row_title,_custom_option_row_price,_custom_option_row_sku,_custom_option_row_sort,_super_products_sku,_super_attribute_code,_super_attribute_option,_super_attribute_price_corr
1111,,Furniture,simple,"Furniture/Living Room","Root Catalog",base,,Red,,,50.0000,,Italy,,"2007-08-28 16:25:46",,,,,"The Magento ottoman will impress with its style while it delivers on quality. This piece of living room furniture is built to last with durable solid wood framing, generous padding and plush stain-resistant microfiber upholstery.",,Microfiber,,,,,,0,/o/t/ottoman.jpg,,,,,,,,Ottoman,Ottoman,Ottoman,299.9900,magotto,,,,Ottoman,,,"Block after Info Column",,299.9900,,,0,,"Living Room",,,,,,"With durable solid wood framing, generous padding and plush stain-resistant microfiber upholstery.",/o/t/ottoman.jpg,,,,,1,2,/o/t/ottoman.jpg,,"2008-08-08 14:59:04",ottoman,ottoman.html,4,20.0000,706.0000,0.0000,1,0,0,1,1.0000,1,100.0000,1,1,,1,0,1,0,1,0.0000,1,0,0,1112,,384822,,1114,,,,,,,,,,,,703,/o/t/ottoman.jpg,,0,1,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,bar1234,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
1112,,Furniture,simple,"Some Non Existent Category","Root Catalog",base,,Red,,,50.0000,,Italy,,"2007-08-28 16:29:24",,,,,"This Magento chair features a fun, futuristic design, with fluid curves and gentle angles that follow the shape of the body to enhance ultimate relaxation. It combines a hint of nostalgia with the up-to-date sensibility and function of modern chairs. It is in soft, velvety microfiber.",,Microfiber,,,,,,0,/c/h/chair.jpg,,,,,,,,Chair,Chair,Chair,129.9900,1112,,,,Chair,,,"Block after Info Column",,129.9900,,,0,,"Living Room",,,,,,"Combining a hint of nostalgia with the up-to-date sensibility and function of modern chairs. It is in soft, velvety microfiber.",/c/h/chair.jpg,,,,,1,2,/c/h/chair.jpg,,"2008-08-08 14:59:17",chair,chair.html,4,50.0000,724.0000,0.0000,1,0,0,1,1.0000,1,100.0000,1,1,,1,0,1,0,1,0.0000,1,0,0,1113,,384822,,1114,,,,,,,,,,,,703,/c/h/chair.jpg,,0,1,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,n2610,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

As you can see the only purpose of the second rows (let's call them "slave" rows) for each product is to provide one more cross-selling connection.

The first problem happens when importer will try to process the "master" row of the second product (SKU = 1112). The validation of the row will fail. In our case it will happen because of non-existing category but the reason can be different. The importer will rightfully skip this row and move to "slave" row. This row will pass validation but as the previous row was ignored it will be assigned to earlier product with SKU = 1111. This means the cross-selling connection will be added to wrong product.

This becomes even worse threat for multilingual stores where products can get wrong names, images or other vital attributes.

This pull request is addressing this issue by grouping rows which belong to the same entity and pushing them into the bunch only if all rows of the group passed validation. Also if one row of the group fails validation the rest of the "slave" rows will not be even checked which will insignificantly speed up processing.

If user prefers importer to process the group regardless if it contains invalid rows he can achieve it by adding sku values along with other required values to each "slave" row.

The second bug is somehow alike. It is derived from the fact that importer will split large data sets into chunks (by default of 500 rows) and save them for future processing. The problem is that "slave" rows which are wrapped into the next bunch are ignored and lost.

This pull request is addressing this issue by closing the bunch only if the there are no more "slave" rows left. So each bunch will always start with a "master" row.

Both bugs above are extremely critical and resulted in my complete catalog of 200k SKUs to be messed up, sales loss and significant damage to the business.

[verklov]

@tim-bezhashvyly, thank you for contributing your pull request! The team will look at it and we will get back to you once we have the answer.

[tim-bezhashvyly]

@verklov So you are not merging pull requests directly into your repo, right? Shall I kill mine then or how does it work?

[verklov]

@tim-bezhashvyly, you are right, we are not merging pull requests directly. Anyway, you should not kill your pull request. In any case it serves as a basis for us. The team reviews code contribution and then makes a decision whether to merge it as is or rewrite it being guided by the idea suggested or issue noted. In some cases though, the team agrees to the pull request being submitted and merges it as is.
In any case, when there is a code contribution available from the community member, this is a valuable thing the team can start working with. I hope this clarifies a bit the process we are following.

Meanwhile, this particular pull request was processed by the team and we released the fix in dev79 just yesterday. Thank you very much for your contribution! Looking forward for new bug reports and ideas from you, Tim!