Allow pkg-config to fallback to system libraries

[samschlegel]

Fixes #13472

Setting PKG_CONFIG_LIBDIR to only look at our libraries means that libgit2 won't find libcurl. Due to libgit2#2555 this means it will build without proxy support. By instead setting PKG_CONFIG_PATH we allow our libraries to take precedence, but don't disable inclusion of system libraries as a fallback.

However I'm not on a Mac and can't check that this wouldn't reintroduce #12514, so if someone could check that that would be awesome :)

[samschlegel]

Though I'm also thinking that maybe including libcurl and keeping pkg-config restricted to the bundled libraries may be the better choice based on how other libraries are included

[vtjnash]

Correct, we usually don't want the build to pick up packages from the system as it makes it harder to create reliable binaries (make binary-dist) and consistent build results.

[samschlegel]

Sounds good. I'll do that instead when I get home and throw a PR up.

[tkelman]

what do you mean by "including libcurl" here?

[samschlegel]

@tkelman I was thinking adding it the downloaded external dependencies, but looking at how openssl is in the required build tools just for libgit2 maybe it would still make sense to also have libcurl there. Curl is required anyways as one of the methods to download external libraries.

I'll reopen this for further discussion

[samschlegel]

If we don't want to let pkg-config pick up all system libraries, and don't want to download and build libcurl, perhaps we could add a whitelist to pull in certain .pc files we would want to pickup from the system

[tkelman]

We still need to do something about proxies here. I'd rather not build libcurl, I think allowing libgit2 to find it is worth the rare issues with fink or macport's iconv.

[vtjnash]

It would also break make binary-dist. libgit2 needs to declare and build all of it's dependencies just like all of the other deps that we have, it doesn't get a free pass.

[tkelman]

No, you're wrong there, read make binary-dist more carefully (edit: apparently it's in

julia/deps/Makefile

Lines 2046 to 2054 in 003f415

	ifeq ($(OS),Linux)
	@# If we're on linux, copy over libssl and libcrypto for libgit2
	-LIBGIT_LIBS=$$(ldd "$@" | tail -n +2 | awk '{print $$(NF-1)}'); \
	for LIB in libssl libcrypto; do \
	LIB_PATH=$$(echo "$$LIBGIT_LIBS" | grep "$$LIB"); \
	echo "LIB_PATH for $$LIB: $$LIB_PATH"; \
	[ ! -z "$$LIB_PATH" ] && cp -v "$$LIB_PATH" $(build_shlibdir); \
	done
	endif

). We currently do give libgit2 a free pass, we bundle binaries of libssl and libcrypto from the build machine in the linux binaries to make https work. Also see make win-extras which certainly isn't building "all the other deps" from source - if we can use something, unpatched, that is always present or easy to get binaries for, I don't see a need to make our makefile complexity problem any worse.

Openssl is even more of a mess to be building from source due to all the security patches. I don't want to get into that. Once libgit2 0.24.0 is released I do plan on adding a source build of mbedtls for linux, but we still need libcurl for proxy support.

[mogthesprog]

Hi Guys,

I was the OP on issue #14167. Any idea when this fix will be available to clone? if not, could I get some help implementing the workaround please?

Cheers,

Morgan

[tkelman]

This change is still a bit controversial, and a few other steps will be necessary to ensure linux binaries contain libcurl, and reduce the chances of bringing back #12514. For now you'll have to apply this patch locally. Try curl -L https://github.com/JuliaLang/julia/pull/13952.patch | git apply

[vtjnash]

or just set USE_SYSTEM_LIBGIT2=1 in your Make.user file and install sudo apt-get install libgit2-dev. if your goal is to use system libraries, might as well just use them.

(closing because #13472 already covers the issue)

[tkelman]

Unlike curl, libgit2's api is still changing and we have zero guarantees that distributions will have a version packaged that we can use.

[samschlegel]

For example, using libcurl to access HTTP(S) was added in libgit 0.23.0 which still isn't packaged in CentOS/RHEL 6/7