[ITensors] [BUG] Julia v1.11 precompile fails when PackageCompiler.jl is also part of the project

[jtschneider]

Description of bug

I encounter a strange bug when precompiling a project environment where the PackageCompiler.jl package is also installed.
This only happens in the current Julia version 1.11, and does not occur in v1.10.

Minimal code demonstrating the bug or unexpected behavior

$ mkdir test_project

$ cd test_project

$ julia                                                                                                                                                   
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.1 (2024-10-16)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Pkg

julia> Pkg.activate(".")
  Activating project at `~/Documents/Code/test_project`

julia> Pkg.add(["ITensors", "PackageCompiler"])
   Resolving package versions...
    Updating `~/Documents/Code/test_project/Project.toml`
  [9b87118b] + PackageCompiler v2.1.20
    Updating `~/Documents/Code/test_project/Manifest.toml`
  [c27321d9] + Glob v1.3.1
  [9b87118b] + PackageCompiler v2.1.20
  [05181044] + RelocatableFolders v1.0.1
  [6c6a2e73] + Scratch v1.2.1
  [4af54fe1] + LazyArtifacts v1.11.0
Precompiling project...
  ✗ ITensors → ITensorsPackageCompilerExt
  0 dependencies successfully precompiled in 15 seconds. 130 already precompiled.
  1 dependency errored.
  For a report of the errors see `julia> err`. To retry use `pkg> precompile`

The precompilation fails with the following error message:

Error message in v1.11

julia> versioninfo()
Julia Version 1.11.1
Commit 8f5b7ca12ad (2024-10-16 10:53 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 32 × AMD Ryzen 9 7950X 16-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver4)
Threads: 1 default, 0 interactive, 1 GC (on 32 virtual cores)
Environment:
  JULIA_DEPOT_PATH = /home/jan/.julia

julia> Pkg.status()
Status `~/Documents/Code/test_project/Project.toml`
  [9136182c] ITensors v0.6.21
  [9b87118b] PackageCompiler v2.1.20

julia> Pkg.precompile()
Precompiling project...
  ✗ ITensors → ITensorsPackageCompilerExt
  0 dependencies successfully precompiled in 15 seconds. 130 already precompiled.

ERROR: The following 1 direct dependency failed to precompile:

ITensorsPackageCompilerExt 

Failed to precompile ITensorsPackageCompilerExt [5b909bbc-0d5a-54f0-8858-6d117796ad2f] to "/home/jan/.julia/compiled/v1.11/ITensorsPackageCompilerExt/jl_0nA6rd".
┌ Warning: The function `inner` is not implemented for (values of) type `Tuple{ITensors.ITensor, ITensors.ITensor}`;
│ this fallback will disappear in future versions of VectorInterface.jl
└ @ VectorInterface ~/.julia/packages/VectorInterface/1L754/src/fallbacks.jl:196
┌ Warning: The function `zerovector` is not implemented for (values of) type `Tuple{ITensors.ITensor, Float64}`;
│ this fallback will disappear in future versions of VectorInterface.jl
└ @ VectorInterface ~/.julia/packages/VectorInterface/1L754/src/fallbacks.jl:33
┌ Warning: The function `zerovector!` is not implemented for (values of) type `ITensors.ITensor`;
│ this fallback will disappear in future versions of VectorInterface.jl
└ @ VectorInterface ~/.julia/packages/VectorInterface/1L754/src/fallbacks.jl:42
┌ Warning: The function `add!!` is not implemented for (values of) type `Tuple{ITensors.ITensor, ITensors.ITensor, Float64, VectorInterface.One}`;
│ this fallback will disappear in future versions of VectorInterface.jl
└ @ VectorInterface ~/.julia/packages/VectorInterface/1L754/src/fallbacks.jl:163
┌ Warning: The function `scalartype` is not implemented for (values of) type `ITensors.ITensor`;
│ this fallback will disappear in future versions of VectorInterface.jl
└ @ VectorInterface ~/.julia/packages/VectorInterface/1L754/src/fallbacks.jl:20
ERROR: LoadError: ArgumentError: No fallback for applying `scalartype` to (values of) type `Any` could be determined
Stacktrace:
  [1] scalartype(T::Type)
    @ VectorInterface ~/.julia/packages/VectorInterface/1L754/src/fallbacks.jl:24
  [2] scalartype(T::Type)
    @ VectorInterface ~/.julia/packages/VectorInterface/1L754/src/fallbacks.jl:25
  [3] scalartype(x::ITensors.ITensor)
    @ VectorInterface ~/.julia/packages/VectorInterface/1L754/src/interface.jl:17
  [4] promote_add(x::ITensors.ITensor, y::ITensors.ITensor, α::Float64, β::VectorInterface.One)
    @ VectorInterface ~/.julia/packages/VectorInterface/1L754/src/VectorInterface.jl:27
  [5] add!!(y::ITensors.ITensor, x::ITensors.ITensor, α::Float64, β::VectorInterface.One)
    @ VectorInterface ~/.julia/packages/VectorInterface/1L754/src/fallbacks.jl:175
  [6] add!!(y::ITensors.ITensor, x::ITensors.ITensor, α::Float64)
    @ VectorInterface ~/.julia/packages/VectorInterface/1L754/src/interface.jl:141
  [7] initialize(iter::KrylovKit.LanczosIterator{ITensors.ITensorMPS.ProjMPO, ITensors.ITensor, KrylovKit.ModifiedGramSchmidt2}; verbosity::Int64)
    @ KrylovKit ~/.julia/packages/KrylovKit/Y0zG7/src/factorizations/lanczos.jl:184
  [8] eigsolve(A::ITensors.ITensorMPS.ProjMPO, x₀::ITensors.ITensor, howmany::Int64, which::Symbol, alg::KrylovKit.Lanczos{KrylovKit.ModifiedGramSchmidt2, Float64}; alg_rrule::KrylovKit.Arnoldi{KrylovKit.ModifiedGramSchmidt2, Float64})
    @ KrylovKit ~/.julia/packages/KrylovKit/Y0zG7/src/eigsolve/lanczos.jl:16
  [9] eigsolve(f::ITensors.ITensorMPS.ProjMPO, x₀::ITensors.ITensor, howmany::Int64, which::Symbol; kwargs::@Kwargs{ishermitian::Bool, tol::Float64, krylovdim::Int64, maxiter::Int64})
    @ KrylovKit ~/.julia/packages/KrylovKit/Y0zG7/src/eigsolve/eigsolve.jl:218
 [10] macro expansion
    @ ~/.julia/packages/ITensors/fUsvl/src/lib/ITensorMPS/src/dmrg.jl:239 [inlined]
 [11] macro expansion
    @ ~/.julia/packages/TimerOutputs/NRdsv/src/TimerOutput.jl:253 [inlined]
 [12] macro expansion
    @ ~/.julia/packages/ITensors/fUsvl/src/lib/ITensorMPS/src/dmrg.jl:238 [inlined]
 [13] macro expansion
    @ ./timing.jl:421 [inlined]
 [14] dmrg(PH::ITensors.ITensorMPS.ProjMPO, psi0::ITensors.ITensorMPS.MPS, sweeps::ITensors.ITensorMPS.Sweeps; which_decomp::Nothing, svd_alg::Nothing, observer::ITensors.ITensorMPS.NoObserver, outputlevel::Int64, write_when_maxdim_exceeds::Nothing, write_path::String, eigsolve_tol::Float64, eigsolve_krylovdim::Int64, eigsolve_maxiter::Int64, eigsolve_verbosity::Int64, eigsolve_which_eigenvalue::Symbol, ishermitian::Bool)
    @ ITensors.ITensorMPS ~/.julia/packages/ITensors/fUsvl/src/lib/ITensorMPS/src/dmrg.jl:206
 [15] dmrg
    @ ~/.julia/packages/ITensors/fUsvl/src/lib/ITensorMPS/src/dmrg.jl:158 [inlined]
 [16] dmrg(H::ITensors.ITensorMPS.MPO, psi0::ITensors.ITensorMPS.MPS, sweeps::ITensors.ITensorMPS.Sweeps; kwargs::@Kwargs{outputlevel::Int64})
    @ ITensors.ITensorMPS ~/.julia/packages/ITensors/fUsvl/src/lib/ITensorMPS/src/dmrg.jl:28
 [17] dmrg
    @ ~/.julia/packages/ITensors/fUsvl/src/lib/ITensorMPS/src/dmrg.jl:21 [inlined]
 [18] #dmrg#515
    @ ~/.julia/packages/ITensors/fUsvl/src/lib/ITensorMPS/src/dmrg.jl:388 [inlined]
 [19] dmrg
    @ ~/.julia/packages/ITensors/fUsvl/src/lib/ITensorMPS/src/dmrg.jl:378 [inlined]
 [20] main(; N::Int64, dmrg_kwargs::@NamedTuple{nsweeps::Int64, maxdim::Int64, cutoff::Float64})
    @ ITensorsPackageCompilerExt ~/.julia/packages/ITensors/fUsvl/ext/ITensorsPackageCompilerExt/precompile_itensors.jl:23
 [21] top-level scope
    @ ~/.julia/packages/ITensors/fUsvl/ext/ITensorsPackageCompilerExt/precompile_itensors.jl:28
 [22] include(mod::Module, _path::String)
    @ Base ./Base.jl:557
 [23] include(x::String)
    @ ITensorsPackageCompilerExt ~/.julia/packages/ITensors/fUsvl/ext/ITensorsPackageCompilerExt/ITensorsPackageCompilerExt.jl:1
 [24] top-level scope
    @ ~/.julia/packages/ITensors/fUsvl/ext/ITensorsPackageCompilerExt/ITensorsPackageCompilerExt.jl:3
 [25] include
    @ ./Base.jl:557 [inlined]
 [26] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
    @ Base ./loading.jl:2790
 [27] top-level scope
    @ stdin:5
in expression starting at /home/jan/.julia/packages/ITensors/fUsvl/ext/ITensorsPackageCompilerExt/precompile_itensors.jl:28
in expression starting at /home/jan/.julia/packages/ITensors/fUsvl/ext/ITensorsPackageCompilerExt/ITensorsPackageCompilerExt.jl:1
in expression starting at stdin:

julia> Pkg.rm("PackageCompiler")
    Updating `~/Documents/Code/test_project/Project.toml`
  [9b87118b] - PackageCompiler v2.1.20
    Updating `~/Documents/Code/test_project/Manifest.toml`
  [c27321d9] - Glob v1.3.1
  [9b87118b] - PackageCompiler v2.1.20
  [05181044] - RelocatableFolders v1.0.1
  [6c6a2e73] - Scratch v1.2.1
  [4af54fe1] - LazyArtifacts v1.11.0

julia> Pkg.precompile()

julia> using ITensors

julia> 

Expected output or behavior

I would expect the same behaviour as in Julia v1.10:

Output of minimal runnable code that *works*

> $ julia +1.10                                                                                                                                             
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.5 (2024-08-27)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> versioninfo()
Julia Version 1.10.5
Commit 6f3fdf7b362 (2024-08-27 14:19 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 32 × AMD Ryzen 9 7950X 16-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 32 virtual cores)
Environment:
  JULIA_DEPOT_PATH = /home/jan/.julia

julia> Pkg.activate(".")
  Activating project at `~/Documents/Code/test_project`

julia> Pkg.resolve()
  No Changes to `~/Documents/Code/test_project/Project.toml`
    Updating `~/Documents/Code/test_project/Manifest.toml`
  [0dad84c5] ↓ ArgTools v1.1.2 ⇒ v1.1.1
  [56f22d72] ~ Artifacts v1.11.0 ⇒ 
  [2a0f44e3] ~ Base64 v1.11.0 ⇒ 
  [ade2ca70] ~ Dates v1.11.0 ⇒ 
  [8ba89e20] ~ Distributed v1.11.0 ⇒ 
  [7b1f6079] ~ FileWatching v1.11.0 ⇒ 
  [9fa8497b] ~ Future v1.11.0 ⇒ 
  [b77e0a4c] ~ InteractiveUtils v1.11.0 ⇒ 
  [4af54fe1] ~ LazyArtifacts v1.11.0 ⇒ 
  [76f85450] ~ LibGit2 v1.11.0 ⇒ 
  [8f399da3] ~ Libdl v1.11.0 ⇒ 
  [37e2e46d] ~ LinearAlgebra v1.11.0 ⇒ 
  [56ddb016] ~ Logging v1.11.0 ⇒ 
  [d6f4376e] ~ Markdown v1.11.0 ⇒ 
  [44cfe95a] ↓ Pkg v1.11.0 ⇒ v1.10.0
  [de0858da] ~ Printf v1.11.0 ⇒ 
  [3fa0cd96] + REPL
  [9a3f8284] ~ Random v1.11.0 ⇒ 
  [9e88b42a] ~ Serialization v1.11.0 ⇒ 
  [6462fe0b] ~ Sockets v1.11.0 ⇒ 
  [2f01184e] ↓ SparseArrays v1.11.0 ⇒ v1.10.0
  [8dfed614] ~ Test v1.11.0 ⇒ 
  [cf7118a7] ~ UUIDs v1.11.0 ⇒ 
  [4ec0a83e] ~ Unicode v1.11.0 ⇒ 
  [deac9b47] ↓ LibCURL_jll v8.6.0+0 ⇒ v8.4.0+0
  [e37daf67] ↓ LibGit2_jll v1.7.2+0 ⇒ v1.6.4+0
  [c8ffd9c3] ↓ MbedTLS_jll v2.28.6+0 ⇒ v2.28.2+1
  [14a3606d] ↓ MozillaCACerts_jll v2023.12.12 ⇒ v2023.1.10
  [4536629a] ↓ OpenBLAS_jll v0.3.27+1 ⇒ v0.3.23+4
  [bea87d4a] ↓ SuiteSparse_jll v7.7.0+0 ⇒ v7.2.1+1
  [8e850ede] ↓ nghttp2_jll v1.59.0+0 ⇒ v1.52.0+1

julia> Pkg.status()
Status `~/Documents/Code/test_project/Project.toml`
  [9136182c] ITensors v0.6.21
  [9b87118b] PackageCompiler v2.1.20

julia> Pkg.precompile()

julia> using ITensors

julia> 

[kmp5VT]

@mtfishman I have started looking at this issue. It looks like PackageCompiler needs the scalartype function that is in ITensorVectorInterfaceExt. I am trying to force the package manager to load both extensions using this line in Project.toml

[extensions]
ITensorsPackageCompilerExt = ["PackageCompiler", "VectorInterface"]

But this does not fix the problem.
I also tried adding using VectorInterface: VectorInterface to the ITensorsPackageCompilerExt/precompile_itensors.jl file and that too didn't load the ITensorVectorInterfaceExt module.
What did work was specifically including the VectorInterface extension file like this
include("$(@__DIR__)/../ITensorsVectorInterfaceExt/ITensorsVectorInterfaceExt.jl") to ITensorsPackageCompilerExt/precompile_itensors.jl