Loading precompiled package fails if a former transitive dependency is not present

[tkelman]

If a transitive dependency was present when a module was last precompiled, but gets removed due to being no longer required, the dependent module fails to load. Example:

~/Julia$ rm -rf jlpkgtmp && JULIA_PKGDIR=$PWD/jlpkgtmp julia-0.6/julia --depwarn=no
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.0-pre.beta.11 (2017-04-03 14:43 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit daefda4* (0 days old master)
|__/                   |  x86_64-linux-gnu

julia> Pkg.init(); Pkg.add("StatsModels")
INFO: Initializing package repository /home/tkelman/Julia/jlpkgtmp/v0.6
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
INFO: Cloning cache of CategoricalArrays from https://github.com/JuliaData/CategoricalArrays.jl.git
INFO: Cloning cache of Compat from https://github.com/JuliaLang/Compat.jl.git
INFO: Cloning cache of DataStructures from https://github.com/JuliaCollections/DataStructures.jl.git
INFO: Cloning cache of DataTables from https://github.com/JuliaData/DataTables.jl.git
INFO: Cloning cache of FileIO from https://github.com/JuliaIO/FileIO.jl.git
INFO: Cloning cache of GZip from https://github.com/JuliaIO/GZip.jl.git
INFO: Cloning cache of NullableArrays from https://github.com/JuliaStats/NullableArrays.jl.git
INFO: Cloning cache of Reexport from https://github.com/simonster/Reexport.jl.git
INFO: Cloning cache of SortingAlgorithms from https://github.com/JuliaCollections/SortingAlgorithms.jl.git
INFO: Cloning cache of SpecialFunctions from https://github.com/JuliaMath/SpecialFunctions.jl.git
INFO: Cloning cache of StatsBase from https://github.com/JuliaStats/StatsBase.jl.git
INFO: Cloning cache of StatsModels from https://github.com/JuliaStats/StatsModels.jl.git
INFO: Installing CategoricalArrays v0.1.3
INFO: Installing Compat v0.21.0
INFO: Installing DataStructures v0.5.3
INFO: Installing DataTables v0.0.2
INFO: Installing FileIO v0.3.1
INFO: Installing GZip v0.3.0
INFO: Installing NullableArrays v0.1.0
INFO: Installing Reexport v0.0.3
INFO: Installing SortingAlgorithms v0.1.1
INFO: Installing SpecialFunctions v0.1.1
INFO: Installing StatsBase v0.13.1
INFO: Installing StatsModels v0.0.2
INFO: Package database updated

julia> using StatsModels
INFO: Precompiling module StatsModels.
<some unrelated method overwrite warnings>

julia> exit()
~/Julia$ JULIA_PKGDIR=$PWD/jlpkgtmp julia-0.6/julia --depwarn=no
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.0-pre.beta.11 (2017-04-03 14:43 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit daefda4* (0 days old master)
|__/                   |  x86_64-linux-gnu

julia> Pkg.pin("StatsBase", v"0.13.0") # does not depend on SpecialFunctions
INFO: Creating StatsBase branch pinned.36e5b015.tmp
INFO: Removing SpecialFunctions v0.1.1

julia> using StatsModels # this should trigger a recompile, not fail
ERROR: ArgumentError: Module SpecialFunctions not found in current path.
Run `Pkg.add("SpecialFunctions")` to install the SpecialFunctions package.
Stacktrace:
 [1] require(::Symbol) at ./loading.jl:403
 [2] _include_from_serialized(::String) at ./loading.jl:157
 [3] _require_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at ./loading.jl:194
 [4] _require_search_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at ./loading.jl:224
 [5] require(::Symbol) at ./loading.jl:409

julia> touch(Pkg.dir("StatsModels", "src", "StatsModels.jl"));

julia> using StatsModels
INFO: Recompiling stale cache file /home/tkelman/Julia/jlpkgtmp/lib/v0.6/StatsModels.ji for module StatsModels.
<some unrelated method overwrite warnings>

[tkelman]

cc @stevengj @vtjnash any thoughts on how to fix this?

[tkelman]

Why do transitive dependencies get saved in and loaded in the top-level ji file, instead of checking via the intermediate dependencies? In the example above, if StatsBase's timestamp was checked first and it had been recompiled, rather than trying to load SpecialFunctions directly from the top-level StatsModels.ji file, this would work fine. This causes a lot of issues on PkgEval.

[stevengj]

@tkelman, that behavior goes way back, to #8745 I think...

[tkelman]

Not saying it's new, but it's always failed to handle this situation correctly. People can't refactor to reduce or replace the number of dependencies they have without potentially breaking all the .ji files for packages downstream.

[tkelman]

How long the behavior has been wrong also doesn't answer the question of why it's doing that. The logic in loading.jl is tough to follow - why is it saving transitive dependencies into the ji file then trying to load things that no longer exist, instead of first checking its direct dependencies?

[stevengj]

I'm just saying that since it traces back to @vtjnash's original precompile patch, he's maybe the one who knows why it is that way. I don't know, and I agree that offhand it seems more reasonable to store immediate dependencies. But the logic of that code is hard to follow.