Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Please allow to use external packages for all dependencies (ex. zstd, flatbuffers, etc) to facilitate packaging #3668

Closed
yurivict opened this issue Feb 16, 2019 · 7 comments

Comments

@yurivict
Copy link

yurivict commented Feb 16, 2019

Here: https://github.com/apache/arrow/blob/master/cpp/CMakeLists.txt#L218

Based on Repology, zstd packages are available for about every OS: https://repology.org/metapackage/zstd/versions

@wesm
Copy link
Member

wesm commented Feb 16, 2019

hi @yurivict what are you asking for exactly? If the user does not have zstd installed, our build system builds the package from source. We should ensure that different kinds of system-level installed packages are properly detected in https://github.com/apache/arrow/blob/master/cpp/cmake_modules/FindZSTD.cmake, though

@yurivict
Copy link
Author

There should be a switch to use only external packages, ex. USE_EXTERNAL_PACKAGES cmake variable.

My use case: I would like to create a FreeBSD port for arrow. On FreeBSD (as in most systems) package builder doesn't allow ad-hoc file downloads started from the project build process. This is for security reasons. So there should a central switch that uses only external packages.

If the user does not have zstd installed, our build system builds the package from source.

Users work on some OSes, which nowadays all have packaging systems (maybe except Windows). Packages should be used. And users shouldn't be building individual softwares themselves, they should use pre-built packages provided by their systems. You are trying to cover up the lack of dependency packages when the right way is to have them created.

@yurivict yurivict changed the title Please use zstd package instead of ExternalProject_Add Please allow to use external packages for all dependencies (ex. zstd, flatbuffers, etc) to facilitate packaging Feb 17, 2019
@wesm
Copy link
Member

wesm commented Feb 17, 2019

Can you please open a JIRA issue? There is also a mailing list thread about improvements to the build system to make packaging for Linux distributions easier (the same concerns apply to FreeBSD)

https://lists.apache.org/thread.html/56e3e5e44901a4ff4f62adc93bb0d34d4c4eac48c993146172e7684e@%3Cdev.arrow.apache.org%3E

You are trying to cover up the lack of dependency packages when the right way is to have them created.

I think this statement is a little bit judgmental -- we are absolutely not trying to "cover up" anything. Since Arrow is still a relatively new project, we have been concerned with providing a uniform development environment to all developers of the C++ project to help promote forward library development progress. In that we have been objectively successful

We don't have enough maintainers to handle support for all of the different third party versions of system libraries on different OS package managers. We have the ExternalProjects set up so that developers are not held back by old packages in their OS PM. This is where we need your help

@yurivict
Copy link
Author

@yurivict
Copy link
Author

This is where we need your help

I am only a volunteer FreeBSD port maintainer. I identify software that needs to be ported, create ports, and later maintain them. If a particular package is not ready to be ported and requires significant efforts, I can't invest too much time into it, because then I'll become a developer on a large number of software packages. I simply don't have enough resources for this, sorry!

Basically, cmake provides all the tools to depend on the external packages. These are cmake scripts and pkg-config instructions. You should just use these tools, as documented in the cmake documentation.

@wesm
Copy link
Member

wesm commented Feb 17, 2019

You should just use these tools, as documented in the cmake documentation.

I agree, though I take issue with the "should" and "just" words here (see http://www.humansystemsinaction.com/lullaby-language/). There is development to do here, and testing. We will also probably need to document which distros have adequate system-level dependencies to build the project, since not all of them do.

@wesm wesm closed this as completed Feb 17, 2019
@kou
Copy link
Member

kou commented Feb 17, 2019

Thanks for opening the JIRA issue.
I'm working of this issue to improve MinGW package (and .deb packages) such as #3675. 0.13.0 or 0.14.0 will support more system libraries. You may be able to work on FreeBSD port with 0.13.0 or 0.14.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants