Skip to content
This repository has been archived by the owner on Aug 16, 2023. It is now read-only.

Wrong download-path for FFTW? #182

Closed
BeastyBlacksmith opened this issue Sep 18, 2019 · 12 comments
Closed

Wrong download-path for FFTW? #182

BeastyBlacksmith opened this issue Sep 18, 2019 · 12 comments

Comments

@BeastyBlacksmith
Copy link

running ]build FFTW in a clean environment brought me this error:

Error building `FFTW`: 
│ [ Info: Downloading https://github.com/JuliaBinaryWrappers/FFTW_jll.jl/releases/download/FFTW-v3.3.9+0/FFTW.v3.3.9.x86_64-linux-gnu.tar.gz to /usr/home/christ/.julia/packages/FFTW/xi4tZ/deps/usr/downloads/FFTW.v3.3.9.x86_64-linux-gnu.tar.gz...
│ ERROR: LoadError: LoadError: Could not download https://github.com/JuliaBinaryWrappers/FFTW_jll.jl/releases/download/FFTW-v3.3.9+0/FFTW.v3.3.9.x86_64-linux-gnu.tar.gz to /usr/home/christ/.julia/packages/FFTW/xi4tZ/deps/usr/downloads/FFTW.v3.3.9.x86_64-linux-gnu.tar.gz:
│ ErrorException("")

This seems to me as if it tries to store the binary in the .julia/packages folder which is read-only AFAIK.

julia> versioninfo()
Julia Version 1.1.1
Commit 55e36cc308 (2019-05-16 04:10 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, ivybridge)
Environment:
  JULIA_EDITOR = atom  -a
  JULIA_NUM_THREADS = 4
@fredrikekre
Copy link
Member

the .julia/packages folder which is read-only AFAIK.

No.

@BeastyBlacksmith
Copy link
Author

Any idea what could be the issue then?

@fredrikekre
Copy link
Member

No, the url works for me.

@stevengj
Copy link
Contributor

(Moved from JuliaMath/FFTW.jl#111)

@stevengj
Copy link
Contributor

Just some kind of proxy or firewall problem? Do other packages based on BinaryBuilder build for you, e.g. ZMQ?

@BeastyBlacksmith
Copy link
Author

Seems this was only a transient issue, since it works today

@BeastyBlacksmith
Copy link
Author

Seems to be connected to shared environments.
Consider the following:

(v1.1) pkg> build FFTW
  Building Conda → `~/.julia/packages/Conda/kLXeC/deps/build.log`
  Building FFTW ─→ `~/.julia/packages/FFTW/2okGQ/deps/build.log`

julia> using Pkg

julia> Pkg.activate("dev",shared=true)
"/usr/home/christ/.julia/environments/dev/Project.toml"

(dev) pkg> add FFTW
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
  Updating `/usr/people/home/christ/.julia/environments/dev/Project.toml`
  [7a1cc6ca] + FFTW v1.0.1
  Updating `/usr/people/home/christ/.julia/environments/dev/Manifest.toml`
 [no changes]

(dev) pkg> build FFTW
  Building Conda → `~/.julia/packages/Conda/kLXeC/deps/build.log`
  Building FFTW ─→ `~/.julia/packages/FFTW/MJ7kl/deps/build.log`
┌ Error: Error building `FFTW`: 
│ [ Info: Downloading https://github.com/JuliaBinaryWrappers/FFTW_jll.jl/releases/download/FFTW-v3.3.9+2/FFTW.v3.3.9.x86_64-linux-gnu.tar.gz to /usr/home/christ/.julia/packages/FFTW/MJ7kl/deps/usr/downloads/FFTW.v3.3.9.x86_64-linux-gnu.tar.gz...
│ ERROR: LoadError: LoadError: Could not download https://github.com/JuliaBinaryWrappers/FFTW_jll.jl/releases/download/FFTW-v3.3.9+2/FFTW.v3.3.9.x86_64-linux-gnu.tar.gz to /usr/home/christ/.julia/packages/FFTW/MJ7kl/deps/usr/downloads/FFTW.v3.3.9.x86_64-linux-gnu.tar.gz:
│ ErrorException("")
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] #download#89(::Bool, ::Function, ::String, ::String) at /usr/home/christ/.julia/packages/BinaryProvider/TcAwt/src/PlatformEngines.jl:498
│  [3] #download at ./none:0 [inlined]
│  [4] #download_verify#90(::Bool, ::Bool, ::Bool, ::Function, ::String, ::String, ::String) at /usr/home/christ/.julia/packages/BinaryProvider/TcAwt/src/PlatformEngines.jl:567
│  [5] #download_verify at ./none:0 [inlined]
│  [6] #install#129(::Prefix, ::String, ::Bool, ::Bool, ::Bool, ::Function, ::String, ::String) at /usr/home/christ/.julia/packages/BinaryProvider/TcAwt/src/Prefix.jl:314
│  [7] (::getfield(BinaryProvider, Symbol("#kw##install")))(::NamedTuple{(:prefix, :force, :verbose),Tuple{Prefix,Bool,Bool}}, ::typeof(install), ::String, ::String) at ./none:0
│  [8] top-level scope at /usr/home/christ/.julia/packages/FFTW/MJ7kl/deps/build_fftw.jl:45
│  [9] include at ./boot.jl:326 [inlined]
│  [10] include_relative(::Module, ::String) at ./loading.jl:1038
│  [11] include(::Module, ::String) at ./sysimg.jl:29
│  [12] include(::String) at ./client.jl:403
│  [13] top-level scope at /usr/home/christ/.julia/packages/FFTW/MJ7kl/deps/build.jl:49
│  [14] include at ./boot.jl:326 [inlined]
│  [15] include_relative(::Module, ::String) at ./loading.jl:1038
│  [16] include(::Module, ::String) at ./sysimg.jl:29
│  [17] include(::String) at ./client.jl:403
│  [18] top-level scope at none:0
│ in expression starting at /usr/home/christ/.julia/packages/FFTW/MJ7kl/deps/build_fftw.jl:43
│ in expression starting at /usr/home/christ/.julia/packages/FFTW/MJ7kl/deps/build.jl:20
│ [16:04:32] curl: symbol lookup error: curl: undefined symbol: curl_url_cleanup
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:1075

@BeastyBlacksmith
Copy link
Author

after running into a similar issue with FFMPEG I think it is a curl issue.
I can trigger that symbol lookup error simply by running something like

shell> curl --version
curl: symbol lookup error: curl: undefined symbol: curl_url_cleanup

Running this in the system shell is fine though. Also using HTTP.download instead of BinaryBuilder.download works for downloading at the REPL trying to actually run ]build FFMPEG when I changed BinaryBuilder.download to use HTTP.download resulted in the following error:



(v1.1) pkg> build FFMPEG

┌ Warning: Dependency graph not a DAG, linearizing anyway

└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:992

  Building MbedTLS → `~/.julia/packages/MbedTLS/X4xar/deps/build.log`

 Resolving package versions...

┌ Error: Error building `MbedTLS`: 

│ ERROR: LoadError: LibraryProduct(nothing, ["libmbedx509"], :libmbedx509, "Prefix(/usr/home/christ/.julia/packages/MbedTLS/X4xar/deps/usr)") is not satisfied, cannot generate deps.jl!

│ Stacktrace:

│  [1] error(::String) at ./error.jl:33

│  [2] #write_deps_file#167(::Bool, ::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /usr/home/christ/.julia/dev/BinaryProvider/src/Products.jl:419

│  [3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0

│  [4] top-level scope at none:0

│  [5] include at ./boot.jl:326 [inlined]

│  [6] include_relative(::Module, ::String) at ./loading.jl:1038

│  [7] include(::Module, ::String) at ./sysimg.jl:29

│  [8] include(::String) at ./client.jl:403

│  [9] top-level scope at none:0

│ in expression starting at /usr/home/christ/.julia/packages/MbedTLS/X4xar/deps/build.jl:72

└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:1075

  Building FFMPEG ─→ `~/.julia/packages/FFMPEG/bkWgb/deps/build.log`

 Resolving package versions...

┌ Error: Error building `FFMPEG`: 

│ ┌ Info: Downloading

│ │   source = "https://github.com/JuliaPackaging/Yggdrasil/releases/download/Bzip2-v1.0.6-2/build_Bzip2.v1.0.6.jl"

│ │   dest = "/usr/home/christ/.julia/packages/FFMPEG/bkWgb/deps/build_Bzip2.v1.0.6.jl"

│ │   progress = NaN

│ │   time_taken = "0.29 s"

│ │   time_remaining = "NaN s"

│ │   average_speed = "2.033 KiB/s"

│ │   downloaded = "612 bytes"

│ │   remaining = "∞ B"

│ └   total = "∞ B"

│ ┌ Info: Downloading

│ │   source = "https://github.com/JuliaPackaging/Yggdrasil/releases/download/Bzip2-v1.0.6-2/build_Bzip2.v1.0.6.jl"

│ │   dest = "/usr/home/christ/.julia/packages/FFMPEG/bkWgb/deps/build_Bzip2.v1.0.6.jl"

│ │   progress = 1.0

│ │   time_taken = "0.0 s"

│ │   time_remaining = "0.0 s"

│ │   average_speed = "1.700 MiB/s"

│ │   downloaded = "3.481 KiB"

│ │   remaining = "0 bytes"

│ └   total = "3.481 KiB"

│ ┌ Info: Downloading

│ │   source = "https://github.com/JuliaPackaging/Yggdrasil/releases/download/Bzip2-v1.0.6-2/Bzip2.v1.0.6.x86_64-linux-gnu.tar.gz"

│ │   dest = "/usr/home/christ/.julia/packages/FFMPEG/bkWgb/deps/usr/downloads/Bzip2.v1.0.6.x86_64-linux-gnu.tar.gz"

│ │   progress = NaN

│ │   time_taken = "0.0 s"

│ │   time_remaining = "NaN s"

│ │   average_speed = "306.152 KiB/s"

│ │   downloaded = "627 bytes"

│ │   remaining = "∞ B"

│ └   total = "∞ B"

│ ┌ Info: Downloading

│ │   source = "https://github.com/JuliaPackaging/Yggdrasil/releases/download/Bzip2-v1.0.6-2/Bzip2.v1.0.6.x86_64-linux-gnu.tar.gz"

│ │   dest = "/usr/home/christ/.julia/packages/FFMPEG/bkWgb/deps/usr/downloads/Bzip2.v1.0.6.x86_64-linux-gnu.tar.gz"

│ │   progress = 1.0

│ │   time_taken = "0.44 s"

│ │   time_remaining = "0.0 s"

│ │   average_speed = "1.104 MiB/s"

│ │   downloaded = "503.081 KiB"

│ │   remaining = "0 bytes"

│ └   total = "503.081 KiB"

│ ERROR: LoadError: LoadError: LibraryProduct(nothing, ["libbz2"], :libbzip2, "Prefix(/usr/home/christ/.julia/packages/FFMPEG/bkWgb/deps/usr)") is not satisfied, cannot generate deps.jl!

│ Stacktrace:

│  [1] error(::String) at ./error.jl:33

│  [2] #write_deps_file#167(::Bool, ::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /usr/home/christ/.julia/dev/BinaryProvider/src/Products.jl:419

│  [3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0

│  [4] top-level scope at none:0

│  [5] include at ./boot.jl:326 [inlined]

│  [6] include_relative(::Module, ::String) at ./loading.jl:1038

│  [7] include at ./sysimg.jl:29 [inlined]

│  [8] include(::String) at /usr/home/christ/.julia/packages/FFMPEG/bkWgb/deps/build.jl:51

│  [9] top-level scope at /usr/home/christ/.julia/packages/FFMPEG/bkWgb/deps/build.jl:52

│  [10] include at ./boot.jl:326 [inlined]

│  [11] include_relative(::Module, ::String) at ./loading.jl:1038

│  [12] include(::Module, ::String) at ./sysimg.jl:29

│  [13] include(::String) at ./client.jl:403

│  [14] top-level scope at none:0

│ in expression starting at /usr/home/christ/.julia/packages/FFMPEG/bkWgb/deps/build_Bzip2.v1.0.6.jl:48

│ in expression starting at /usr/home/christ/.julia/packages/FFMPEG/bkWgb/deps/build.jl:44

└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:1075

Seeing this and the discussion in JuliaLang/julia#27043 I am curious if it would work with a LibCURL implementation of download.

@BeastyBlacksmith
Copy link
Author

It turned out that there was a libcurl4.5.0 in the lib folder of my julia installation and after deleting this everything was fine.
Thanks to @giordano and @aviks.

@SimonDanisch
Copy link
Member

Could we maybe check this? This is a pretty annoying issue I also run into, which is hard to debug...

@stevengj
Copy link
Contributor

It seems pretty common to have a broken curl in the PATH, which ends up breaking everything… the ultimate resolution seems to be for Julia to either ship its own libcurl or ship with a native download function based on the HTTP module (JuliaLang/julia#27043).

@BeastyBlacksmith
Copy link
Author

Checking the version of the to be used download tool (e.g. in __init__) should be possible meanwhile.

@giordano giordano mentioned this issue Oct 13, 2019
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@SimonDanisch @stevengj @fredrikekre @BeastyBlacksmith