RecursiveDirectoryIterator misbehaves if the directory gets modified while iterating #2903
Comments
caco3
added
bug
|
See also nextcloud/updater#509 (comment) for further information |
|
FYI: Fixed version is in the example code. Guess that should not be the case. ps: Thanks a lot for your work on the Nextcloud/Hostpoint/update issue. I'm not a PHP dev, but pretty sure that using a directory iterator while manipulating the underlying folder is a bad idea. |
|
Iterating on a directory's contents while you are actively modifying its contents feels risky just on principle. It would seem alright to automatically "snapshot" one particular directory's contents before returning its data for iteration, but that also means not being able to pick up any changes to the directory - desirable or not. Then the recursive iteration is harder: if you were recursing on All in all, the safest course of action would certainly be to make your own snapshot before performing any actions (like with iterator_to_array), then doing any necessary last-minute sanity checks before making each change, but I wouldn't yet discount there being something that could be done here. |
Thanks, I corrected the example.
Thanks, hopefully they will accept my fix. Currently it looks like they do not want to accept it because they officially do not support BSD, see nextcloud/updater#510 (comment) @damianwadley Thanks for your comment. Oddly, therere are many examples on the web doing it without $iterator = new DirectoryIterator($dir);
foreach ($iterator as $fileinfo) { .. }and foreach (new DirectoryIterator($dir) as $fileinfo) { .. }? |
Read it and I think they worded it badly and/or got it wrong. But their motivation to understand and fix the underlying issue is sound. And in this case, once it can be shown that there is no (vanilla) documentation stating that their usage of the iterators is save, then I am sure they will accept a fix and everyone, including users on their supported platforms, wins. |
Would flip the implied question: Where can one find authoritative documentation stating that it is save to keep using a DirectoryIterator instance once its content got manipulated? Any chance a PHP developer can help us out here? |
|
Dear all Thanks for your valuable comments and your support on this! I did more tests: Original:foreach (getRecursiveDirectoryIterator("src") as $path) {
[..]
-> 522: src/1000 -> dst/1000 -> badIntermediate variable:$iter = getRecursiveDirectoryIterator("src");
foreach ($iter as $path) {
[..]
-> 522: src/1000 -> dst/1000 -> badFile list:$fileList = iterator_to_array(getRecursiveDirectoryIterator("src"), true);
foreach ($fileList as $path => $fileInfo) {
[..]
-> 1000: src/1000 -> dst/1000 -> okThis answers my question, an intermediate variable ($iter) does not contain the filelist but the iteration object: print_r($iter);
/*RecursiveIteratorIterator Object
()*/where as the filelist is a pure array: print_r($fileList);
/*Array
(
[src/1] => SplFileInfo Object
(
[pathName:SplFileInfo:private] => src/1
[fileName:SplFileInfo:private] => 1
)
[src/2] => SplFileInfo Object
(
[pathName:SplFileInfo:private] => src/2
[fileName:SplFileInfo:private] => 2
)
[..]*/So the usage of |
|
@caco3 can you test if the same issue also happens with $i = 0;
mkdir("dst");
$handle = opendir("src");
while (false !== ($entry = readdir($handle))) {
$i++;
$path = "src/$entry";
$newPath = "dst/$entry";
echo("$i: $path -> $newPath\n");
rename($path, $newPath);
}
closedir($handle);
echo("EOF"); |
|
Yes, same issue: |
|
I found an answer to a similar question that suggests that this behavior undefined and depends on the operating system and file system: https://stackoverflow.com/a/39017355/1031606 So i’d guess that PHP is not at fault here, it just behaves the same way as the operating/file system it’s dealing with does. There is also an article from Apple that explicitly states that this is not supported on their HFS file system: https://web.archive.org/web/20220122122948/https://support.apple.com/kb/TA21420?locale=en_US |
The "fault" of PHP is that it does not document the guarantees (not) given. At least I could not find any documentation on this topic. |
|
I agree that it just is a lack of documentation. IMO a critical one. |
If you're satisfied with this being a documentation problem, I can move it to the appropriate place: |
|
Yes, please go ahead |
@caco3 IIRC you weren't able to reproduce this on FreeBSD stock, but only in the hoster's environment, correct? NFS was the other known variable: |
|
@caco3 does the following behave exactly the same way for you? function getRecursiveDirectoryIterator(?string $folder = null): \RecursiveIteratorIterator {
return new \RecursiveIteratorIterator(new \RecursiveDirectoryIterator($folder, \RecursiveDirectoryIterator::SKIP_DOTS), \RecursiveIteratorIterator::CHILD_FIRST);
}
$i = 0;
mkdir("dst");
$iterator = getRecursiveDirectoryIterator("src");
$iterator->rewind();
while ($iterator->valid()) {
$path = $iterator->current();
$i++;
$newPath = str_replace("src", "dst", $path);
echo("$i: $path -> $newPath\n");
rename($path, $newPath); // <- makes the iterator to break
$iterator->next();
}
echo("EOF\n"); |
Honestly I can't remember and don't have the local FreeBSD available anymore.
I tested it on my hosters system and yes, still same issue: |
Description
I am trying to run a simple PHP script on the FreeBSD 12.4 system of my web hoster (hostpoint.ch).
It seems that there is some non-standard behaviour in it.
Preparations
setup a src directory with a lot of files:
create a php script using following content:
Expected output
It is expected that all 500 files get moved from
srctodst:Effective Output
The iterator has 2 glitches, files 95..188 and 285..380 get skipped:
Note that changing the iterator to create a list and looping through the list works around this issue:
PHP Version
7.4, 8.0, 8.1
Operating System
FreeBSD 12.4
The text was updated successfully, but these errors were encountered: