Many automated builds/tests are broken with a libgit2 ECERTIFICATE error #13399

simonster · 2015-10-01T04:40:03Z

The errors are along the lines of:

ERROR: LoadError: GitError(Code:ECERTIFICATE, Class:SSL, The SSL certificate is invalid)
 [inlined code] from libgit2/error.jl:96
 in clone at libgit2/repository.jl:95
 in clone at libgit2.jl:303
 in anonymous at /home/ubuntu/buildbot/slave/coverage_ubuntu14_04-x64/build/julia-1ed51ea4cd/share/julia/test/libgit2.jl:72
 in temp_dir at /home/ubuntu/buildbot/slave/coverage_ubuntu14_04-x64/build/julia-1ed51ea4cd/share/julia/test/libgit2.jl:62
 in temp_dir at /home/ubuntu/buildbot/slave/coverage_ubuntu14_04-x64/build/julia-1ed51ea4cd/share/julia/test/libgit2.jl:5

@staticfloat first pointed this out on a buildbot, but it's also happening on Travis:
https://travis-ci.org/JuliaStats/NullableArrays.jl/jobs/83049793
https://travis-ci.org/JuliaLang/JLD.jl/jobs/83031348
https://travis-ci.org/JuliaStats/PDMats.jl/jobs/83032826

The text was updated successfully, but these errors were encountered:

wildart · 2015-10-01T05:22:17Z

We could disable certificate check, at least for METADATA packages. For clone, let's prompt user with warning and allow to cancel a started operation.

tkelman · 2015-10-01T06:15:33Z

Wouldn't that largely defeat the purpose of using https in the first place? It looks like we need to configure libgit2 and openssl in the nightly binaries to bundle its own working certs.

staticfloat · 2015-10-01T06:45:13Z

If we bundle ca certs somewhere does that fix the problem?

tkelman · 2015-10-01T07:05:15Z

I would hope so, but not sure the best way to test that. It looks like we don't run the tests on the centos 5 binary builders right now, would first want to check if they pass the libgit2 and pkg tests locally. Then figure out how to make them self contained and work when moved to different clean machines.

vks · 2015-10-01T12:17:36Z

I also get this error when trying to install packages with nightly.

julia> Pkg.add("Gadfly")
INFO: Initializing package repository ~/.julia/v0.5
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
ERROR: GitError(Code:ECERTIFICATE, Class:SSL, The SSL certificate is invalid)
 [inlined code] from libgit2/error.jl:96
 in clone at libgit2/repository.jl:95
 in clone at libgit2.jl:303
 in anonymous at pkg/dir.jl:57
 in cd at file.jl:22
 in init at pkg/dir.jl:55
 in cd at pkg/dir.jl:28
 in add at pkg.jl:27

wildart · 2015-10-01T20:14:25Z

We could require along with libssl dependency to install ca-certificates from openssl bundle (or whatever alternative for particular platform). That should be easy to fix.

tkelman · 2015-10-01T20:50:15Z

@wildart this is about tarball binaries. We want to avoid requiring manual installation of any packages for those. We should try including the certificates in the binary, if that can be made to work. @staticfloat little help finding out if the centos5 builders are passing the https tests locally?

staticfloat · 2015-10-01T20:57:55Z

Looks like they pass;

$ ./julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.0-dev+564 (2015-10-01 19:15 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit 966850c (0 days old master)
|__/                   |  x86_64-unknown-linux-gnu

julia> Base.runtests("libgit2")
     * libgit2               in   7.75 seconds
    SUCCESS

tkelman · 2015-10-01T21:02:40Z

Great, thanks. Now where do the certs live and how do we make the tarball build see them? (Also what is the license on them?)

wildart · 2015-10-01T21:13:42Z

MPL 2.0 from Mozilla, however they need to be converted to proper format.

staticfloat · 2015-10-01T21:17:36Z

I don't think the answer here is to bundle our own cert.pem bundle; that way lies madness, incompatibility and security holes. The proper way to do this is to point our libssl.so to the system-provided certificates.

On Centos5, the SSL master ca bundle is stored in /etc/pki/tls, and you can see via strace that Julia is trying to access that when you run the libgit2 tests:

open("/etc/pki/tls/cert.pem", O_RDONLY) = -1 ENOENT (No such file or directory)

On Ubuntu, on the other hand, the directory holding this kind of stuff is /usr/lib/ssl; you can get these things via openssl version -d, I'm currently looking into how we can override this search path at runtime.

tkelman · 2015-10-01T21:22:07Z

Is this baked into openssl, or could a build flag on libgit2 make it search more places?

Do arch and opensuse and whatever other distros all use different paths for this? Is it lib64 in some cases?

staticfloat · 2015-10-01T21:22:53Z

It's a little bit of a mess: http://gagravarr.org/writing/openssl-certs/others.shtml#ca-openssl

wildart · 2015-10-01T21:24:43Z

There is a libgit2 flag GIT_OPT_SET_SSL_CERT_LOCATIONS which points to cert file location.

staticfloat · 2015-10-01T21:30:13Z

That looks good; if someone can give me an example of how I would set that option (like in this example) I can test it out and figure out what works on Ubuntu. Note that we would want to set the path parameter of that option on Ubuntu, but the file parameter of that option on CentOS5. I'm thinking we'll just build a small database of files/paths to search, and then choose the first we isfile() as true.

wildart · 2015-10-01T21:51:04Z

cert_file = "/cert/file/location/cacert.pem"
GIT_OPT_SET_SSL_CERT_LOCATIONS = 12
err = ccall((:git_libgit2_opts, :libgit2), Cint,
             (Cint, Cstring, Ptr{Void}),
              GIT_OPT_SET_SSL_CERT_LOCATIONS, cert_file, C_NULL)
err == 0 && LibGit2.clone("https://github.com/JuliaLang/Example.jl.git", "/tmp/Example")
# LibGit2.Error.last_error() # check error

staticfloat · 2015-10-01T22:23:49Z

cert_path = "/usr/lib/ssl/certs"
GIT_OPT_SET_SSL_CERT_LOCATIONS = 12
err = ccall((:git_libgit2_opts, :libgit2), Cint,
                    (Cint, Ptr{Void}, Cstring),
                     GIT_OPT_SET_SSL_CERT_LOCATIONS, C_NULL, cert_path)
LibGit2.clone("https://github.com/JuliaLang/Example.jl.git", "/tmp/Example")

On Ubuntu, works! In short, you're a scholar and a gentleman.

tkelman · 2015-10-01T22:25:23Z

Great. Little convoluted to have to hard-code these and run through them at LibGit2 module init time, but sounds worth trying. We should probably spin up a set of docker containers of different distros that we know people use Julia on and check that our list covers them all.

staticfloat · 2015-10-01T22:26:50Z

What do you think of querying openssl version -d first, to see if we can shortcut the whole process?

tkelman · 2015-10-01T22:36:15Z

How likely are common base distros to have working certs present, but not command-line openssl? And depending how long shelling out or querying the presence of a bunch of files takes, maybe we do this on the first invocation of a Pkg command that needs it, rather than on every Julia startup?

wildart · 2015-10-01T22:52:26Z

I am not sure that certificates will be present, in fresh OS installation, unless you install them explicitly.

wildart · 2015-10-01T22:57:00Z

Why don't set certificate location through ENV flag and check it on Julia start up?

staticfloat · 2015-10-01T22:57:13Z

How likely are common base distros to have working certs present, but not command-line openssl?

Not very likely, I don't think. I have yet to login to a *nix computer that doesn't have openssl installed, including locked-down cluster-type ones. (EDIT: nevermind, I found one; my openwrt router doesn't have it installed by default)

I am not sure that certificates will be present, in fresh OS installation, unless you install them explicitly.

SSL certificate authority files are pretty fundamental to using the internet; unless you are talking about a computer that is not intended to use the internet, (in which case this issue is somewhat moot) they will have certificate files. They may be outdated or restricted but they will have them.

wildart · 2015-10-01T22:59:47Z

~~Certificates are only required for secure connection which is optional. Browsers usually come with their own certs.~~ Well, OS package managers rely on them. So, I guess we need to keep cert locations for every distro. Not looking good. I vote for ENV variable.

pao · 2015-10-02T02:08:46Z

openwrt router doesn't have it installed by default

Anyone willing to get Julia running on MIPS is welcome to deal with that themselves.

lkuper · 2015-10-02T18:33:43Z

This issue is breaking the Travis build of one of our Julia packages, too (only on the Linux build job, not OS X). Is there a workaround?

lkuper · 2015-10-06T18:59:53Z

@staticfloat I've just pulled master (which seemed to pull in a lot of libgit2 stuff) and rebuilt, and I'm seeing the following:

$ ./julia 
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.0-dev+640 (2015-10-06 18:24 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit 128f8c6 (0 days old master)
|__/                   |  x86_64-redhat-linux

julia> Pkg.update()
INFO: Updating METADATA...
WARNING: fetch: GitError(Code:ERROR, Class:Net, SSL error: error:140E0114:SSL routines:SSL_shutdown:uninitialized)

Is there something I'm doing wrong -- perhaps some missing env var or dependency?

Edit: I'm seeing the above issue on both Ubuntu 14.04.3 LTS, and CentOS release 6.6.

staticfloat · 2015-10-06T19:00:46Z

I've never seen that particular error before. What system are you running on?
-E

lkuper · 2015-10-06T19:19:49Z

@staticfloat Oops, just edited my comment. This is on both Ubuntu and CentOS.

staticfloat · 2015-10-06T19:29:17Z

I just tried it out on Ubuntu 14.04 and I had no problem. Do you have libssl-dev installed? (You can check with dpkg -l | grep libssl-dev on Ubuntu, yum list openssl-devel on CentOS). After ensuring that you have the development libraries for SSL installed, can you try make -C deps distclean-libgit2 followed by a make again and see if that changes things?

wildart · 2015-10-06T20:40:39Z

SSL_shutdown error happens when there is no connection. The message is misleading because there is a bug in handling SSL connection in libgit2. It is already fixed in upstream, #3445.

lkuper · 2015-10-06T21:55:03Z

@staticfloat @wildart I do have the necessary libraries on both platforms. (On CentOS the package is called openssl-devel.) I tried make -C deps distclean-libgit2 && make, and in both cases I got the same error as above when I ran Pkg.update().

I tried blowing away my ~/.julia/v0.5 directory and calling Pkg.init() again, and got a similar error:

julia> Pkg.init()
INFO: Initializing package repository /home/lkuper/.julia/v0.5
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
ERROR: GitError(Code:ERROR, Class:Net, SSL error: error:140E0114:SSL routines:SSL_shutdown:uninitialized)
 [inlined code] from libgit2/error.jl:96
 in clone at libgit2/repository.jl:185
 in clone at libgit2.jl:303
 in anonymous at pkg/dir.jl:49
 in cd at file.jl:22
 in init at pkg/dir.jl:47
 in init at pkg.jl:23

I'm a bit mystified, but I'll see if make clean helps and I'll try to figure out if anything is wrong with my git configuration.

lkuper · 2015-10-06T23:05:49Z

@staticfloat Same problem persists after a make distcleanall. By the way, I'm behind a firewall and I use git config --global url."https://".insteadOf git://. I also have Julia 0.3.11 installed, and there, I have no problem with Pkg.update() or Pkg.add(). So it doesn't seem like the problem is with my git configuration. Could libgit2 be not seeing my ~/.gitconfig for some reason?

wildart · 2015-10-06T23:40:24Z

@lkuper Can you try a libgit2 master build? This SSL_shutdown error was fixed week ago. I suspect that something happens before this error and incorrect SSL shutdown overwrites cause of your problem.

lkuper · 2015-10-07T00:18:12Z

@wildart Thank you for trying to help. Yes, libgit2 builds and installs for me with no problems, following the directions at https://github.com/libgit2/libgit2#building-libgit2---using-cmake . julia/deps/libgit2/build$ cmake --build . also works fine. This doesn't solve the issue in Julia, though. I'll file a separate bug.

waTeim · 2016-02-09T06:35:07Z

Whelp, I have this problem now, what do?

julia> Pkg.init();
INFO: Initializing package repository /home/jeffw/.julia/v0.5
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
ERROR: GitError(Code:ECERTIFICATE, Class:SSL, The SSL certificate is invalid)

julia> versioninfo()
Julia Version 0.5.0-dev+2336
Commit 9cda53d (2016-01-28 14:53 UTC)
Platform Info:
  System: Linux (arm-linux-gnueabihf)     
  CPU: ARMv7 Processor rev 10 (v7l)        <----- yes, it's ARM
  WORD_SIZE: 32
  BLAS: libopenblas (NO_AFFINITY ARMV7)
  LAPACK: libopenblas
  LIBM: libm
  LLVM: libLLVM-3.7.1

Cert location

Lots of certs in the mozilla directory.

ls /usr/share/ca-certificates/
./           ../          cacert.org/  mozilla/     spi-inc.org/

tkelman · 2016-02-09T06:41:37Z

Source build or nightly binary?

waTeim · 2016-02-09T06:51:29Z

nightly binary

tkelman · 2016-02-09T07:05:01Z

Ok looks like the default location fix in #13429 only works when binaries are built from a centos-like system (which we do for x86 and amd64), not an ubuntu-like system (which we do for arm). I don't think we have full control over the arm buildbot, though @ViralBShah and/or @staticfloat may know more of the details. As a workaround you may be able to try setting LibGit2.set_ssl_cert_locations to the locations that are right for your system?

waTeim · 2016-02-09T07:12:29Z

Sure I can try that

julia> methods(LibGit2.set_ssl_cert_locations)
# 1 method for generic function "set_ssl_cert_locations":
set_ssl_cert_locations(cert_file) at libgit2.jl:502

Does this need to be called once per cert or something?

waTeim · 2016-02-09T18:47:29Z

I tried both that function and the above for including an entire directory. Still no luck

function set_git_path(cert_path)
   GIT_OPT_SET_SSL_CERT_LOCATIONS = 12
   err = ccall((:git_libgit2_opts,:libgit2),Cint,(Cint,Ptr{Void},Cstring),GIT_OPT_SET_SSL_CERT_LOCATIONS,C_NULL,cert_path)
end

Followed by:

julia> include("julia-git.jl")
set_git_path (generic function with 1 method)

julia> set_git_path("/etc/ssl/certs")
0

julia> Pkg.init()
INFO: Initializing package repository /home/jeffw/.julia/v0.5
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
ERROR: GitError(Code:ECERTIFICATE, Class:SSL, The SSL certificate is invalid)

Maybe they're actually invalid? This linux distro is older (2013) but maybe they have updated their certs? Is there a way to tell?

tkelman · 2016-02-09T18:54:42Z

Which distro? Some have separate ca-certificates vs ca-certificates-mozilla packages but I'm not sure how they differ. Might be worth a new issue, closed issues aren't the best place for active debugging.

waTeim · 2016-02-09T19:22:48Z

That's true, but I wasn't sure this was worth a new issue or if this had not already been dealt with or just "closed for now." or what.

This distro is Yocto Linux, poky release.

This did work: (I guess you just have to arbitrarily know which is the right file):

julia> LibGit2.set_ssl_cert_locations("/etc/ssl/certs/ca-certificates.crt")
0

julia> Pkg.init()
INFO: Initializing package repository /home/jeffw/.julia/v0.5
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl

success

julia> Pkg.add("StateSpace")
ERROR: Base.Pkg.PkgError("unknown package StateSpace")

julia> Pkg.clone("git://github.com/ElOceanografo/StateSpace.jl.git");
INFO: Cloning StateSpace from git://github.com/ElOceanografo/StateSpace.jl.git
INFO: Computing changes...
INFO: Cloning cache of ArrayViews from https://github.com/JuliaLang/ArrayViews.jl.git
INFO: Cloning cache of Calculus from https://github.com/johnmyleswhite/Calculus.jl.git
INFO: Cloning cache of Compat from https://github.com/JuliaLang/Compat.jl.git
INFO: Cloning cache of Distributions from https://github.com/JuliaStats/Distributions.jl.git
INFO: Cloning cache of ForwardDiff from https://github.com/JuliaDiff/ForwardDiff.jl.git
INFO: Cloning cache of NaNMath from https://github.com/mlubin/NaNMath.jl.git
INFO: Cloning cache of PDMats from https://github.com/JuliaStats/PDMats.jl.git
INFO: Cloning cache of StatsBase from https://github.com/JuliaStats/StatsBase.jl.git
INFO: Cloning cache of StatsFuns from https://github.com/JuliaStats/StatsFuns.jl.git
INFO: Installing ArrayViews v0.6.4
INFO: Installing Calculus v0.1.14
INFO: Installing Compat v0.7.9
INFO: Installing Distributions v0.8.9
INFO: Installing ForwardDiff v0.1.5
INFO: Installing NaNMath v0.2.0
INFO: Installing PDMats v0.3.6
INFO: Installing StatsBase v0.7.4
INFO: Installing StatsFuns v0.2.0

to use to point libgit2 to specific bundle of trusted CA certificates. Relates to: JuliaLang#13399, JuliaLang#15128

pclausen · 2018-02-25T03:11:40Z

On Windows this https://stackoverflow.com/questions/48950748/julia-git-error helped me on Windows 7.

Summary:

Close Julia window
Go to https://support.microsoft.com/en-us/help/3140245/update-to-enable-tls-1-1-and-tls-1-2-as-a-default-secure-protocols-in
Use the "Easy Fix"
Download and run
Now re-run Julia --> Pkg.update() works!

UltraInstinct05 · 2018-07-10T08:41:53Z

I have run the easy fix but the same error keeps coming back even if I re run julia. I am using a 64 bit machine, Windows 7 OS & running Julia 0.6.3. Julia.exe is also in the path.
What can I do ?

hbtmgz · 2018-08-19T05:35:04Z

I have met the same problem and I run the easy fix , while it would not work

simonster added bug Indicates an unexpected problem or unintended behavior packages Package management and loading labels Oct 1, 2015

wildart self-assigned this Oct 1, 2015

yuyichao mentioned this issue Oct 1, 2015

Make functors macros hygienic and works outside NumericFuncs module. lindahua/NumericFuns.jl#6

Merged

staticfloat closed this as completed in #13429 Oct 3, 2015

lkuper mentioned this issue Oct 7, 2015

libcurl problems with libgit2 on master #13472

Closed

Ismael-VC mentioned this issue Feb 18, 2016

[ARM] GitError The SSL certificate is invalid from nightlies. #15128

Closed

wildart mentioned this issue Mar 13, 2016

Support SSL_CERT_FILE and SSL_CERT_DIR env vars #15485

Merged

wildart added a commit to wildart/julia that referenced this issue Mar 13, 2016

Use OpenSSL environment variables SSL_CERT_FILE and SSL_CERT_DIR

2a59c5f

to use to point libgit2 to specific bundle of trusted CA certificates. Relates to: JuliaLang#13399, JuliaLang#15128

nalimilan mentioned this issue Dec 15, 2017

Julia master: Error in testset bunchkaufman: SSL certificate is invalid #25096

Closed

mbargull mentioned this issue Jan 30, 2018

MentaLiST: Backport libxml2 fix to v0.1.x series bioconda/bioconda-recipes#7336

Closed

5 tasks

dfornika mentioned this issue Feb 8, 2018

Build failures due to libgit2/SSL error conda-forge/julia-feedstock#45

Closed

Naetmul mentioned this issue Feb 26, 2018

Error when adding packaging on julia #26167

Closed

staticfloat mentioned this issue Mar 29, 2019

Get MbedTLS to use system trusted roots JuliaLang/MbedTLS.jl#200

Closed

Many automated builds/tests are broken with a libgit2 ECERTIFICATE error #13399

Many automated builds/tests are broken with a libgit2 ECERTIFICATE error #13399

Comments

simonster commented Oct 1, 2015

wildart commented Oct 1, 2015

tkelman commented Oct 1, 2015

staticfloat commented Oct 1, 2015

tkelman commented Oct 1, 2015

vks commented Oct 1, 2015

wildart commented Oct 1, 2015

tkelman commented Oct 1, 2015

staticfloat commented Oct 1, 2015

tkelman commented Oct 1, 2015

wildart commented Oct 1, 2015

staticfloat commented Oct 1, 2015

tkelman commented Oct 1, 2015

staticfloat commented Oct 1, 2015

wildart commented Oct 1, 2015

staticfloat commented Oct 1, 2015

wildart commented Oct 1, 2015

staticfloat commented Oct 1, 2015

tkelman commented Oct 1, 2015

staticfloat commented Oct 1, 2015

tkelman commented Oct 1, 2015

wildart commented Oct 1, 2015

wildart commented Oct 1, 2015

staticfloat commented Oct 1, 2015

wildart commented Oct 1, 2015

pao commented Oct 2, 2015

lkuper commented Oct 2, 2015

lkuper commented Oct 6, 2015

staticfloat commented Oct 6, 2015

lkuper commented Oct 6, 2015

staticfloat commented Oct 6, 2015

wildart commented Oct 6, 2015

lkuper commented Oct 6, 2015

lkuper commented Oct 6, 2015

wildart commented Oct 6, 2015

lkuper commented Oct 7, 2015

waTeim commented Feb 9, 2016

Cert location

tkelman commented Feb 9, 2016

waTeim commented Feb 9, 2016

tkelman commented Feb 9, 2016

waTeim commented Feb 9, 2016

waTeim commented Feb 9, 2016

tkelman commented Feb 9, 2016

waTeim commented Feb 9, 2016

pclausen commented Feb 25, 2018

UltraInstinct05 commented Jul 10, 2018

hbtmgz commented Aug 19, 2018