Unable to install packages when Package directory is placed on a Network Drive

[bmharsha]

Issue is, I'm encountering an error whenever I use Pkg.add() or Pkg.clone() in Julia.

Here is my configuration

• Windows Environment variable JULIA_PKGDIR is set to a folder on Network Drive (Y:\jtemp)
• Operating System = Windows 2008 R2 server
• Julia is installed on a Network Drive

julia> Pkg.dir()
"Y:\\insjulia\\v0.4"

julia> Base.LOAD_CACHE_PATH
2-element Array{ByteString,1}:
 "Y:\\insjulia\\lib\\v0.4"
 "Y:\\Julia-0.4.2-pre\\usr\\lib\\julia"

Exact error message when Pkg.add() is executed

julia> Pkg.add("JDBC")
INFO: Cloning cache of Compat from git://github.com/JuliaLang/Compat.jl.git
fatal: could not create leading directories of 'Y:\insjulia\v0.4\.cache\Compat'
ERROR: failed process: Process(`git clone -q --mirror git://github.com/JuliaLang
/Compat.jl.git 'Y:\insjulia\v0.4\.cache\Compat'`, ProcessExited(128)) [128]
 [inlined code] from error.jl:22
 in pipeline_error at process.jl:555
 in sync_end at task.jl:413
 [inlined code] from task.jl:422
 in add at pkg/entry.jl:64
 in add at pkg/entry.jl:73
 [inlined code] from promotion.jl:228
 in anonymous at pkg/dir.jl:31
 in cd at file.jl:32
 in cd at pkg/dir.jl:31
 [inlined code] from essentials.jl:112
 in add at pkg.jl:23

julia> Pkg.add("Gadfly")
ERROR: SystemError: mkdir: File exists
 [inlined code] from strings/io.jl:43
 in mkdir at file.jl:42
 in sync_end at task.jl:413
 [inlined code] from task.jl:422
 in add at pkg/entry.jl:64
 in add at pkg/entry.jl:73
 [inlined code] from promotion.jl:228
 in anonymous at pkg/dir.jl:31
 in cd at file.jl:32
 in cd at pkg/dir.jl:31
 [inlined code] from essentials.jl:112
 in add at pkg.jl:23

Error message when Pkg.clone() is executed

julia> Pkg.clone("https://github.com/JuliaDB/JDBC.jl.git")
INFO: Cloning JDBC from https://github.com/JuliaDB/JDBC.jl.git
INFO: Computing changes...
ERROR: SystemError: mkdir: File exists
 [inlined code] from strings/io.jl:43
 in mkdir at file.jl:42

 julia> Pkg.clone("https://github.com/dcjones/Gadfly.jl.git")
INFO: Cloning Gadfly from https://github.com/dcjones/Gadfly.jl.git
INFO: Computing changes...
ERROR: SystemError: mkdir: File exists
 [inlined code] from strings/io.jl:43
 in mkdir at file.jl:42

Additional Information

• Pkg.init() gets executed successfully, it downloads the Metadata and creates files & folders on Network drive, so the program has write access to Network drive and this even rules out the possibility of network issues.
• I tested the aforementioned configuration on following Julia binaries (without any success)

Julia version 0.4.1 (64 bit)
Julia 0.5.0-bae49b97a8-win64.exe (Inspired by following issue #12166)
Julia 0.4.2 Pre-release binaries 64 bit (#14186)

[tkelman]

The "could not create leading directories error message" is coming from https://github.com/git-for-windows/git/blob/104a5fedf701cf75dbba874d91c63b60766a73b1/builtin/clone.c#L903, and from what I can see the single quotes look like part of the output formatting.

Something in this function https://github.com/git-for-windows/git/blob/104a5fedf701cf75dbba874d91c63b60766a73b1/sha1_file.c#L112-L162 is failing. Unfortunately that error message isn't preserving what the return value was from safe_create_leading_directories_const. I wonder if the git-for-windows binaries are built with debugging symbols so I could try stepping into the git call and see which failure case was happening.

[tkelman]

I was able to reproduce this. I think it's due to NTFS junctions not working properly on network drives. When the target is a directory we default to trying to create a junction in the symlink function. While Vista and newer support true symlinks which would probably work on network drives, you need either admin access or to modify a default permissions setting to be able to create them. With the change from #14225 made locally, the "could not create leading directories" error goes away. The attempted junction creation must've been confusing git - the cache link was acting strangely, sometimes returning all zeros from stat or disappearing from windows explorer on a refresh of the directory.

If we had a way of identifying network drives, then maybe we could not set the junction flag in that case. In #12166 it sounded like we'd need to add statfs functionality to libuv for that though.

Unfortunately with the "could not create leading directories" error fixed, I now get errors that look a lot like #12166 (comment)

julia> Pkg.add("Dates")
INFO: Cloning cache of Dates from git://github.com/quinnj/Dates.jl.git
INFO: Installing Dates v0.4.4
error: internal error: refs/remotes/origin/deprecated is not a valid packed refe
rence!
error: internal error: refs/remotes/origin/master is not a valid packed referenc
e!
fatal: update_ref failed for ref 'HEAD': cannot update the ref 'HEAD': Trying to
 write ref refs/heads/master with nonexistent object 3ccb8b94938518f75b789cd794f
495066482a2b7
fatal: The remote end hung up unexpectedly

Still looking into this, but this might be an upstream git bug.

Separately, we should probably make our SystemErrors more descriptive from mkdir and other functions, listing the file they were trying to operate on.

[tkelman]

Reported the "not a valid packed reference!" issue at git-for-windows/git#554, let's see what happens there.

edit: I found a workaround, adding the file:// prefix seems to fix this.

[tkelman]

Should be fixed (in 0.4.2 by 906288e, on master by #14225). Leave a comment or open a new issue if not.