1.6: Installed dependencies not detected with LOAD_PATH == ["@"]

[fonsp]

Julia 1.6.0-beta1, not an issue on Julia 1.5

Dependencies not loading

If LOAD_PATH is set to ["@"], then installed dependencies are not always picked up by require. Here is a MWE, but be warned that it deletes the .julia/compiled/v1.6 folder to simulate a clean setup:

# 1. Delete the .julia/compiled/v1.6 folder
let
    d = joinpath(DEPOT_PATH[1], "compiled", "v1.6") # change to julia version
    isdir(d) && rm(d, recursive=true)
end


# 2. Create an empty Pkg Context and add HTTP
import Pkg
ctx = Pkg.Types.Context(env=Pkg.Types.EnvCache(
    joinpath(mktempdir(),"Project.toml")
));
Pkg.add(ctx, [Pkg.PackageSpec(name="HTTP")])


# 3. Set LOAD_PATH to ["@"]
empty!(LOAD_PATH)
push!(LOAD_PATH, "@")


# 4. Set the Pkg environment as the active project
Base.ACTIVE_PROJECT[] = dirname(ctx.env.project_file)


# 5. Import HTTP
using HTTP

The output:

julia> using HTTP
[ Info: Precompiling HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3]
ERROR: LoadError: ArgumentError: Package MbedTLS_jll [c8ffd9c3-330d-5841-b78e-0817d7145fa1] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
 [1] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:986
 [2] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:910
 [3] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:897
 [4] include
   @ ./Base.jl:386 [inlined]
 [5] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
   @ Base ./loading.jl:1209
 [6] top-level scope
   @ none:1
 [7] eval
   @ ./boot.jl:360 [inlined]
 [8] eval(x::Expr)
   @ Base.MainInclude ./client.jl:446
 [9] top-level scope
   @ none:1
in expression starting at /Users/fons/.julia/packages/MbedTLS/4YY6E/src/MbedTLS.jl:1
ERROR: LoadError: LoadError: Failed to precompile MbedTLS [739be429-bea8-5141-9913-cc70e7f3736d] to /Users/fons/.julia/compiled/v1.6/MbedTLS/jl_FQyCu9.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::Base.TTY, internal_stdout::Base.TTY)
    @ Base ./loading.jl:1356
  [3] compilecache(pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1302
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1017
  [5] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:910
  [6] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:897
  [7] include(mod::Module, _path::String)
    @ Base ./Base.jl:386
  [8] include(x::String)
    @ HTTP ~/.julia/packages/HTTP/cxgat/src/HTTP.jl:1
  [9] top-level scope
    @ ~/.julia/packages/HTTP/cxgat/src/HTTP.jl:18
 [10] include
    @ ./Base.jl:386 [inlined]
 [11] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1209
 [12] top-level scope
    @ none:1
 [13] eval
    @ ./boot.jl:360 [inlined]
 [14] eval(x::Expr)
    @ Base.MainInclude ./client.jl:446
 [15] top-level scope
    @ none:1
in expression starting at /Users/fons/.julia/packages/HTTP/cxgat/src/IOExtras.jl:1
in expression starting at /Users/fons/.julia/packages/HTTP/cxgat/src/HTTP.jl:1
ERROR: Failed to precompile HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3] to /Users/fons/.julia/compiled/v1.6/HTTP/jl_twgigx.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::Base.TTY, internal_stdout::Base.TTY)
   @ Base ./loading.jl:1356
 [3] compilecache(pkg::Base.PkgId, path::String)
   @ Base ./loading.jl:1302
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1017
 [5] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:910
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:897

Fixed by adding @stdlib to LOAD_PATH

If we add @stdlib to LOAD_PATH:

# [insert 1.-3. from above]

# 3b. Add @stdlib
push!(LOAD_PATH, "@stdlib")

# [insert 4.-5. from above]

then it works!

Pkg.instantiate hangs

If I add a Pkg.instantiate call, like the error suggest, I encounter a new problem: the precompilation stalls at 2/5:

# [insert 1.-4. from above]


# 4b. Instantiate
Pkg.instantiate(ctx)


# 5. Import HTTP
using HTTP

output:

julia> Pkg.instantiate(ctx)
Precompiling project...
  Progress [================>                        ]  2/5

(It's stuck until I interrupt with Ctrl+C.)

Context

As part of fonsp/Pluto.jl#844, I was setting the LOAD_PATH inside notebooks to ["@"] (instead of the default ["@", "@v#.#", "@stdlib"]), to isolate the notebook's package environment from the global env (@v#.#). Since I add stdlibs to the Project.toml automatically, I did not include @stdlib in the LOAD_PATH. I now added @stdlib to the LOAD_PATH to fix my problem, but I am reporting this bug in case it is useful.

[KristofferC]

Should be fixed by JuliaLang/julia#39572. Should be in RC2 (coming in a day or two). For now, you can also add @stdlib to LOAD_PATH to work around it.

[fredrikekre]

Can you open a new issue for the

Pkg.instantiate hangs

part?