Memory leaks on bulk objects save

[FredoVelcro-zz]

We are importing lots of objects (more than 100000) and the server memory keep increasing.
We tried many things to localize the origin of the problem, and it seems to be in the save() method/process itself.

What we tried to free memory :

• unset the objects vars upon completed save() or finish using it
• use \Pimcore::collectGarbage(); every X turns (ex: 100, 500, 1000, 5000 etc)
• use Version::disable(); eahc time before $myObject->save();
• use native calls of gc_* PHP garbage functions

and nothing changed...!
The only thing that prevent the memory growing was...commenting the save() call..

Can you tell us if you did some changes that could affect this since version 6.2.3 ? (DAO ?)

Best regards,

Fred

[dpfaffenbauer]

#3104

[dvesh3]

Long story, short #3104 (comment)

[dpfaffenbauer]

But: what I don't get is, why doesn't that happen with Doctrine ORM? That also uses doctrine/dbal?

[FredoVelcro-zz]

we did not have the problem with Pimcore 6.2.3 ...

[brusch]

@FredoVelcro if possible it would be nice if you can do a profiler comparison between the old version and 6.2.3, imho that's the only way how we can find out what the problem actually is.

[brusch]

@FredoVelcro ping

[NiklasBr]

F.W.I.W. I'm experiencing this with Pimcore 6.6.4 and PHP 7.4, running a CLI script which does this:

        $csvArray = $csv->decode(file_get_contents($filePath), CsvEncoder::FORMAT);
        foreach ($csvArray as $i => $item) {
            if ($i % 11 === 0) {
                \Pimcore::collectGarbage();
            }

            foreach ($item as $levelText => $name) {
                $output->writeln("Processing {$name} at level {$level}");

                $category = new Category();
                // ... set some values
                $category->save();

                $page = new Page();
                // ... set some values
                $page->save();
            }
        }

After about 100 rows memory usage has increased to about 4GB.

[dpfaffenbauer]

1. streaming the csv helps
2. disable doctrine logging helps a lot
3. Memory leak with PDOConnection->prepare() doctrine/dbal#3047 still is the main issue here.

[NiklasBr]

Returning to this issue...

disable doctrine logging helps a lot

I found that explicitly disabling profiling had an even greater impact:

doctrine:
    dbal:
        connections:
            default:
                logging: false
                profiling: false
                ...

[NiklasBr]

@brusch are there plans to work around the DBAL issue in Pimcore 7?

[brusch]

@NiklasBr If there's a way, we can try that (but to be honest, it's currently not on our priority list - PRs however are of course always welcome 😉).
But I really think that main issue should be fixed at the origin (doctrine/dbal#3047)

[NiklasBr]

I wish I knew enough to do it, but at the same time I find it hard to believe that this below kind of use case is not uncommon in Pimcore and it need to be solved somehow:

        $this->processCategoryData($rootCategoryObject);

    // ....

    protected function processCategoryData(Category $category)
    {
        if ($category->hasChildren()) {
            foreach ($category->getChildren() as $childCategory) {
                // ... Process some data on the object ... then save it:
                $childCategory->save();
                $this->processCategoryData($childCategory);
            }
        }
    }

Unless there is another way I can do the same which I have not yet discovered?

[brusch]

To be honest, my knowledge about doctrine/dbal#3047 is also very limited, but I don't see any way to fix that just on Pimcore's end.

[NiklasBr]

Afraid that's so. Do you happen to have any best practices or other workarounds not in this thread (I'm already using them all)?

[brusch]

Not really other than using a message queue with workers.

[NiklasBr]

Unfortunately it still happened when I tried it like that.

[Tsuba]

Hello dears, I'm currently working on Pimcore 6.7 and I get same problems as mentionned about memory leak. Is it something which is fixed in next versions of Pimcore ?

[MagPim]

Same question…

[NiklasBr]

No, it has not improved in Pimcore X, doctrine/dbal is still at 2.x which is the affected version:

pimcore/composer.json

Line 55 in e105061

"doctrine/dbal": "^2.11.0",

Symfony is compatible with 3.x, see for example:

https://github.com/symfony/cache/blob/945bcebfef0aeef105de61843dd14105633ae38f/composer.json#L37

https://github.com/symfony/lock/blob/a78fda52b1b6f74d60e642e91d0e0133b08a8546/composer.json#L25

It is unclear to me why Pimcore did not opt to update. Maybe @brusch knows.

[fashxp]

see #8618

[MagPim]

Thanks for your reponse, even though I don't like it...

[brusch]

@MagPim well yes, we neither, but you can help the guys at Doctrine to speed things up 😉

[dpfaffenbauer]

@brusch doctrine/orm 2.10 is now compatible with dbal 3: https://github.com/doctrine/orm/releases/tag/2.10.0

[brusch]

That's nice, but doesn't help ... doctrine/migrations is still not compatible with 3.0.

[dpfaffenbauer]

🤭 s**t. Will further monitor the progress of migrations and let you know once they update too.

[brusch]

See also doctrine/migrations#1198

[brusch]

Dependency issues are solved, anyone willing to work on that?

[dpfaffenbauer]

I can start with it ;)

[brusch]

@dpfaffenbauer cool, thanks a lot! 👍

[stale]

Thanks a lot for reporting the issue. The issue was not considered by us as "Priority" or "Backlog", so we're not gonna work on that anytime soon. In case this is a bug report, please create a pull request fixing the issue, we'll then review it as soon as possible. If you're interested in contributing a feature, please contact us first here before creating a pull request, we'll then decide whether we'd accept it or not. Thanks for your understanding.

[GeorgII-web]

Could it be related to a #15866 ?

In short, a long running processes needs a manual call for a RuntimeCache::clear(); in a loop.

[fashxp]

should be improved in Pimcore 11.