Rename syscall broken for non-empty directories

[hut8]

Bug

I'm pretty sure rename (2) is broken when moving non-empty directories

Reproduction

mkdir -p test-case/a/b
touch test-case/a/b/awesomesauce.txt
mv test-case/a/b test-case/
# mv: cannot move ‘test-case/a/b’ to ‘test-case/b’: Permission denied

How I encountered this

When running bundle install with gems containing native extensions, the build scripts eventually move the completed .so output into a directory. This fails mysteriously. It's consistent across every ruby gem with native extensions.

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /root/rails-app/vendor/bundle/ruby/2.3.0/gems/byebug-8.2.1/ext/byebug
    /usr/bin/ruby2.3 -r ./siteconf20160412-28335-c0622i.rb extconf.rb
    creating Makefile

current directory: /root/rails-app/vendor/bundle/ruby/2.3.0/gems/byebug-8.2.1/ext/byebug
make "DESTDIR=" clean

current directory: /root/rails-app/vendor/bundle/ruby/2.3.0/gems/byebug-8.2.1/ext/byebug
make "DESTDIR="
compiling breakpoint.c
compiling byebug.c
compiling context.c
compiling locker.c
compiling threads.c
linking shared-object byebug/byebug.so

current directory: /root/rails-app/vendor/bundle/ruby/2.3.0/gems/byebug-8.2.1/ext/byebug
make "DESTDIR=" install
/usr/bin/install -c -m 0755 byebug.so ./.gem.20160412-28335-1lbx7tv/byebug

Permission denied @ rb_file_s_rename - (./.gem.20160412-28335-1lbx7tv/byebug, /root/rails-app/vendor/bundle/ruby/2.3.0/extensions/x86_64-linux/2.3.0/byebug-8.2.1/byebug)

Here's the build directory:

root@localhost:~/rails-app/vendor/bundle/ruby/2.3.0/gems/byebug-8.2.1/ext/byebug# ll
total 476
drwxrwxrwx 2 root root      0 Apr 12 20:19 ./
drwxrwxrwx 2 root root      0 Apr 12 20:19 ../
-rw-r--r-- 1 root root  11770 Apr 12 20:19 breakpoint.c
-rw-rw-rw- 1 root root  56480 Apr 12 20:19 breakpoint.o
-rw-r--r-- 1 root root  20324 Apr 12 20:19 byebug.c
-rw-r--r-- 1 root root   3867 Apr 12 20:19 byebug.h
-rw-rw-rw- 1 root root  86704 Apr 12 20:19 byebug.o
-rwxrwxrwx 1 root root 144652 Apr 12 20:19 byebug.so*
-rw-r--r-- 1 root root  15993 Apr 12 20:19 context.c
-rw-rw-rw- 1 root root  68736 Apr 12 20:19 context.o
-rw-r--r-- 1 root root    503 Apr 12 20:19 extconf.rb
-rw-r--r-- 1 root root   1604 Apr 12 20:19 locker.c
-rw-rw-rw- 1 root root  13696 Apr 12 20:19 locker.o
-rw-rw-rw- 1 root root   7890 Apr 12 20:19 Makefile
-rw-rw-rw- 1 root root      0 Apr 12 20:19 .RUBYARCHDIR.-.byebug.time
-rw-r--r-- 1 root root   4549 Apr 12 20:19 threads.c
-rw-rw-rw- 1 root root  28584 Apr 12 20:19 threads.o

... and the destination directory:

root@localhost:~/rails-app# ls -lha /root/rails-apps/vendor/bundle/ruby/2.3.0/extensions/x86_64-linux/2.3.0/byebug-8.2.1
total 8.0K
drwxrwxrwx 2 root root   0 Apr 12 20:19 .
drwxrwxrwx 2 root root   0 Apr 12 20:36 ..
-rw-rw-rw- 1 root root 881 Apr 12 20:19 gem_make.out

In the build directory, the ./.gem.20160412-28335-1lbx7tv/byebug stuff was deleted as a part of cleaning up (or something), so I ran:

make clean && make && make install

Then I had a reasonable looking build directory, like it would have been right before the rename failed:

root@localhost:~/rails-app/vendor/bundle/ruby/2.3.0/gems/byebug-8.2.1/ext/byebug# ll -R
.:
total 572
drwxrwxrwx 2 root root      0 Apr 12 20:48 ./
drwxrwxrwx 2 root root      0 Apr 12 20:19 ../
-rw-r--r-- 1 root root  11770 Apr 12 20:19 breakpoint.c
-rw-rw-rw- 1 root root  56480 Apr 12 20:47 breakpoint.o
-rw-r--r-- 1 root root  20324 Apr 12 20:19 byebug.c
-rw-r--r-- 1 root root   3867 Apr 12 20:19 byebug.h
-rw-rw-rw- 1 root root  86704 Apr 12 20:47 byebug.o
-rwxrwxrwx 1 root root 144652 Apr 12 20:47 byebug.so*
-rw-r--r-- 1 root root  15993 Apr 12 20:19 context.c
-rw-rw-rw- 1 root root  68736 Apr 12 20:47 context.o
-rw-r--r-- 1 root root    503 Apr 12 20:19 extconf.rb
drwxrwxrwx 2 root root      0 Apr 12 20:48 .gem.20160412-28335-1lbx7tv/
-rw-r--r-- 1 root root   1604 Apr 12 20:19 locker.c
-rw-rw-rw- 1 root root  13696 Apr 12 20:47 locker.o
-rw-rw-rw- 1 root root   7890 Apr 12 20:19 Makefile
-rw-rw-rw- 1 root root      0 Apr 12 20:48 .RUBYARCHDIR.-.byebug.time
-rw-r--r-- 1 root root   4549 Apr 12 20:19 threads.c
-rw-rw-rw- 1 root root  28584 Apr 12 20:47 threads.o

./.gem.20160412-28335-1lbx7tv:
total 4
drwxrwxrwx 2 root root 0 Apr 12 20:48 ./
drwxrwxrwx 2 root root 0 Apr 12 20:48 ../
drwxrwxrwx 2 root root 0 Apr 12 20:48 byebug/

./.gem.20160412-28335-1lbx7tv/byebug:
total 192
drwxrwxrwx 2 root root      0 Apr 12 20:48 ./
drwxrwxrwx 2 root root      0 Apr 12 20:48 ../
-rwxr-xr-x 1 root root 144652 Apr 12 20:48 byebug.so*

Now without Ruby, let me try to do the move step:

root@localhost:~/rails-app/vendor/bundle/ruby/2.3.0/gems/byebug-8.2.1/ext/byebug/.gem.20160412-28335-1lbx7tv# mv byebug/ /root/rails-app/vendor/bundle/ruby/2.3.0/extensions/x86_64-linux/2.3.0/byebug-8.2.1/byebug
mv: cannot move ‘byebug/’ to ‘/root/rails-app/vendor/bundle/ruby/2.3.0/extensions/x86_64-linux/2.3.0/byebug-8.2.1/byebug’: Permission denied

For reference, here's the Ruby call that fails: http://rxr.whitequark.org/mri/source/file.c#2586
although as you can see it happens with mv, so it's not really Ruby's fault.

Here's an strace of my attempt with mv: https://gist.github.com/anonymous/485b87aa511b62542881302a47ab4f13

This is the error that I don't think should be an error:
https://gist.github.com/anonymous/485b87aa511b62542881302a47ab4f13#file-debug-win-L97

[hut8]

More information:

Creating the destination directory that should have been creating during mv works fine:

mkdir /root/rails-app/vendor/bundle/ruby/2.3.0/extensions/x86_64-linux/2.3.0/byebug-8.2.1/byebug

And I can copy the .so file there fine as well.

[russalex]

We have a fix inbound. Not sure when it will show up on insider builds, but it's on the way.

Old:

root@localhost:~# mkdir -p test-case/a/b
root@localhost:~# touch test-case/a/b/awesomesauce.txt
root@localhost:~# mv test-case/a/b test-case/
mv: cannot move ‘test-case/a/b’ to ‘test-case/b’: Permission denied

New:

root@RUSSALEX-DESK:~# mkdir -p test-case/a/b
root@RUSSALEX-DESK:~# touch test-case/a/b/awesomesauce.txt
root@RUSSALEX-DESK:~# mv test-case/a/b test-case/
root@RUSSALEX-DESK:~# 

[russalex]

Additional, looks to be the same as #44.

[hut8]

Thanks so much! This is the coolest thing I've seen in a while; Microsoft has really changed the last couple years.

[karacaenes]

I believe this should also fix #3 #14 and #96 too.

[russalex]

This one looks to be fixed (just tested build 14332). Please feel free to close (or I will in about a week if no other comments).

Glad you like the project. We're having a blast doing this and love the feedback / questions / issues.

[hut8]

I'm downloading that build now! So excited.

[sebastian-greco]

@russalex I don't know if it's the channel to say this... but, great work guys! and most of all thanks having this fixed and pushed so quickly!

[hut8]

@russalex great work indeed! Totally unexpected turnaround time on this. Thanks so much.