Errno: 2; proc_open(): fork failed - Resource temporarily unavailable

[brendt]

I'm trying to run some benchmarks with this package. I'm running 30 iterations of the following:

use Amp\Parallel\Worker\DefaultPool;
use function Amp\ParallelFunctions\parallelMap;
use function Amp\Promise\wait;

// ...

public function runIteration()
{
    $pool = new DefaultPool();

    $promises = parallelMap(range(1, 50), function () {
        sleep(1);

        return 2;
    }, $pool);

    $result = wait($promises);
}

After iteration 7, this exception is always thrown:

Iteration 8:
In Runner.php line 85:

  [Amp\Process\ProcessException]
  Could not start process Errno: 2; proc_open(): fork failed - Resource temporarily unavailable


Exception trace:
 Amp\Process\Internal\Posix\Runner->start() at /Users/brent/dev/spatie/async-benchmark/vendor/amphp/process/lib/Process.php:96
 Amp\Process\Process->start() at /Users/brent/dev/spatie/async-benchmark/vendor/amphp/parallel/lib/Context/Process.php:118
 Amp\Parallel\Context\Process->start() at /Users/brent/dev/spatie/async-benchmark/vendor/amphp/parallel/lib/Worker/AbstractWorker.php:93
 Amp\Parallel\Worker\AbstractWorker->enqueue() at /Users/brent/dev/spatie/async-benchmark/vendor/amphp/parallel/lib/Worker/DefaultPool.php:119
 Amp\Parallel\Worker\DefaultPool->enqueue() at /Users/brent/dev/spatie/async-benchmark/vendor/amphp/parallel-functions/src/functions.php:38
 Amp\ParallelFunctions\{closure}() at n/a:n/a
 array_map() at /Users/brent/dev/spatie/async-benchmark/vendor/amphp/parallel-functions/src/functions.php:56
 Amp\ParallelFunctions\{closure}() at n/a:n/a
 Generator->current() at /Users/brent/dev/spatie/async-benchmark/vendor/amphp/amp/lib/Coroutine.php:39
 Amp\Coroutine->__construct() at /Users/brent/dev/spatie/async-benchmark/vendor/amphp/amp/lib/functions.php:61
 Amp\call() at /Users/brent/dev/spatie/async-benchmark/vendor/amphp/parallel-functions/src/functions.php:63
 Amp\ParallelFunctions\parallelMap() at /Users/brent/dev/spatie/async-benchmark/src/Benchmark1/Amp.php:25
 Spatie\Async\Benchmark\Benchmark1\Amp->runIteration() at /Users/brent/dev/spatie/async-benchmark/src/BenchmarkCommand.php:28
 Spatie\Async\Benchmark\BenchmarkCommand->execute() at /Users/brent/dev/spatie/async-benchmark/vendor/symfony/console/Command/Command.php:252
 Symfony\Component\Console\Command\Command->run() at /Users/brent/dev/spatie/async-benchmark/vendor/symfony/console/Application.php:855
 Symfony\Component\Console\Application->doRunCommand() at /Users/brent/dev/spatie/async-benchmark/vendor/symfony/console/Application.php:233
 Symfony\Component\Console\Application->doRun() at /Users/brent/dev/spatie/async-benchmark/vendor/symfony/console/Application.php:143
 Symfony\Component\Console\Application->run() at /Users/brent/dev/spatie/async-benchmark/console.php:9

benchmark1:amp

[brendt]

Just noticed that by not specifying a DefaultPool, this error doesn't happen.

[kelunik]

I can't reproduce it, can you? Maybe it was just a temporary issue?

[kelunik]

Well, with 30 iterations, I can. With $i < 7 it doesn't happen, with $i < 8 it does happen.

<?php

require __DIR__ . '/../vendor/autoload.php';

use Amp\Parallel\Worker\DefaultPool;
use function Amp\ParallelFunctions\parallelMap;
use function Amp\Promise\wait;

for ($i = 0; $i < 7; $i++) {
    $pool = new DefaultPool();

    $promises = parallelMap(range(1, 50), function () {
        sleep(1);

        return 2;
    }, $pool);

    $result = wait($promises);
}

[kelunik]

This is due to cyclic references, adding unset($pool); gc_collect_cycles(); helps. I solved them for DefaultPool locally now, but WorkerProcess still has cyclic references and isn't cleaned up properly.

[brendt]

Can't seem to reproduce it any longer. Very strange.. Thanks for looking into it!

[kelunik]

cyclic-refs in amphp/parallel partially fixes this. New error:

PHP Fatal error:  Uncaught Amp\Process\ProcessException: Could not start process Errno: 2; proc_open(): unable to create pipe Too many open files in /home/kelunik/GitHub/amphp/parallel-closure/vendor/amphp/process/lib/Internal/Posix/Runner.php:85